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PROBLEM 

^ Within  the  context  of  military  training  system  development,  the  question  of  how 
to  assess  the  cost  versus  expected  benefits  of  innovations  in  training  technology 
has  become  of  major  importance.  Before  this  question  can  be  answered,  it  must  first 
be  demonstrated  that  both  the  cost  and  the  benefits  of  such  innovations  can  be  quan- 
titatively measured.  Then  it  must  be  demonstrated  that  accurate  predictions  can  be 
made  with  regard  to  development  and  operational  costs,  training  effectiveness,  and 
expected  timesavings. 

APPROACH 

Three  separate  activities  were  undertaken  to  solve  this  problem.  The  first 
was  to  define  a descriptive  model  of  a generalized,  computer-based  training  system.  ^ 
This  model  defined  the  domain  of  such  systems  to  be  addressed  by  a predictive  models' 
It  also  identified  the  component  activities  involved  in  such  a training  system, 
isolated  seven  clusters  of  factors  which  would  impact  tl^  operation  of  the  system, 
and  identified  the  relationship  of  each  cluster  to  the  separate  component  activities. 

^he  second  step  was  generation  of  a predictive  model  of  student  performance  in 
a computer-based  training  environment.  The  purpose  of  this  model  was  to  predict 
the  time  required  for  a student  to  complete  an  arbitrary  unit  of  instruction  in  a 
self-paced  training  program  given  only  characteristics  of  the  student,  of  the  content 
taught,  and  of  the  method  employed  to  teach  the  content  in  a conventional,  lock-step 
course. 

The  third  activity  was  to  develop  a computer  program  which  incorporated  the 
predictive  model  and  estimated  the  cost  of  implementing  and  operating  this  particular, 
innovative  training  technique,  i.e.,  conversion  from  conventional  lock-step  to  self- 
paced,  computer-based  instruction.  » 


RESULTS  \ 

A linear  prediction  model  was  developed  to  predict  a s .dent's  time  required  to 
complete  his  first  attempt  on  a given  lesson.  Fourteen  predictor  variables  were 
selected,  eleven  descriptive  of  the  instructional  course  content  and  three  descrip- 
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tive  of  student  characteristics.  The  final  regression  equation  produced  on  R = .63 

accounting  for  39  percent  of  the  variance  in  a student's  first-attempt-lesson  time. 

2 

When  the  regression  equation  was  validated  on  a second  sample,  the  R resulted  in  a 
loss  of  only  5.34  percentage  points  in  accounting  for  the  variance  in  the  criterion 
variable,  first-attempt-lesson  time.  The  basic  model  was  then  modified  to  predict 
total  course  completion  time. 

A computer  program,  "Aid  for  Instructional  Development  and  Evaluation  (AIDE)," 
was  written.  It  is  a modification  of  Rand  Corporation's  MODCOM  cost  analysis  program. 
It  retains  all  of  the  cost-analysis  features  and  outputs  of  MODCOM  while  allowing 
course  completion  time  to  be  estimated  by  the  regression  equation  developed  in  this 
study.  Course  completion  times  and  cost  estimates  were  generated  for  a standard 
sample  and  three  additional  samples  in  which  student  or  course  content  characteris- 
tics were  varied. 

CONCLUSIONS 

This  study  demonstrated  that  a viable  model  can  be  developed  which  predicts 
the  costs  and  timesavings  associated  with  individualization  of  an  arbitrary  training 
course  through  the  use  of  a computer-based  training  system.  The  generality  and  pre- 
cision of  such  a model  is  dependent  on  the  breadth  and  accuracy  of  the  data  base  from 
which  the  model  is  derived.  Although  tne  data  base  employed  in  this  study  was  not 
as  complete  as  could  be  desired,  the  resultant  model  was  found  to  be  reasonably 
accurate  and  is  sufficiently  general  to  be  of  value  in  the  evaluation  of  courses  for 
individualization.  More  importantly,  a methodology  has  been  developed  and  demon- 
strated which  will  become  an  even  more  useful  tool  as  additional  data  are  accumulated 
from  computer-based  training  systems. 

The  results  of  exercising  the  AIDE  program  illustrated  the  impact  of  variations 
in  course  content  on  the  timesavings  to  be  expected  from  individualization.  Current 
methods  of  instruction,  methods  employed  in  the  course  as  it  is  taught  in  conventional, 
lock-step  mode  were  found  to  be  sensitive  indicators  of  variations  in  course  content 
and,  hence,  of  expected  timesavings.  Thus,  performance  oriented  training,  for  which 
the  Plan  of  Instruction  indicates  a relatively  high  proportion  of  demonstration- 
performance  activities,  presents  less  promise  for  individualization  than  do  courses 
containing  a high  proportion  of  cognitive  and,  particularly,  memorization  content. 
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1 .0  INTRODUCTION 

1.1  BACKGROUND 

A question  of  primary  importance  to  those  concerned  with  the  design, 
development,  implementation,  or  evaluation  of  innovations  in  training  technology 
is  one  which  addresses  the  cost  and  effectiveness  of  such  innovations.  Within  the 
context  of  military  training  system  development,  interest  has  recently  been  focused 
on  this  question  as  it  relates  to  innovations  in  the  area  of  computer-based 
training  (CBT).  Here  the  concern  has  been  one  of  not  only  attempting  to  find  means 
for  evaluating  the  actual  benefits  attributable  to  the  innovation  relative  to  its 
cost,  but  also  to  find  a means  for  evaluating  the  expected  benefits.  That  is, 
before  large  sums  of  money  are  invested  in  computer-based  training  innovations,  it 
is  important  to  know  (a)  whether  the  benefits  and  costs  associated  with  the 
innovations  can  be  quantitatively  evaluated  and  (b)  whether  cost-effectiveness 
data  to  support  implementation  decisions  can  be  predicted,  given  quantifiable 
information  on  training  system  parameters  and  their  interactions  with  trainee  and 
training  material  characteristics. 

To  address  the  preceding  concern  adequately,  both  descriptive  and  predi cti ve 
models  of  a computer-based  training  system  are  required.  Descriptive  models 
serve  the  purpose  of  identifying  the  relevant  parameters  affecting  cost  and  effecti- 
veness decisions  and  of  indicating  the  degree  to  which  these  parameters  may  be 
quantified.  Predictive  models  serve  the  purpose  of  calculating  expected  outcomes, 
given  information  on  the  descriptive  parameters  and  the  relationships  between  them. 

A major  problem,  however,  is  that  appropriate  descriptive  and  predictive  models 
have  not  been  sufficiently  integrated  in  an  operational  decision  model  which  can 
be  used  in  planning  cost-effective  implementations  of  CBT  innovations. 

The  present  study  was  undertaken  to  address  this  need.  The  specific  goals  and 
objectives  of  the  effort  are  described  in  the  following  section,  after  which  the 
types  of  problems  involved  in  the  development,  integration,  and  validation  of 
descriptive  and  predictive  CBT  models  are  discussed  in  a literature  review. 
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1.2  GOALS  AND  OBJECTIVES 

The  goals  of  the  present  study  were  threefold:  (1)  to  define  and  describe 

those  computer-based  training  system  parameters  likely  to  affect  cost-effective- 
ness decisions,  i.e.,  to  produce  a descriptive  model  of  a general izable  CBT  system, 
including  all  relevant  instructional,  student,  and  training  system  variables; 

(2)  to  model,  as  accurately  as  possible,  anticipated  student  performance  in  the 
generalized  system  described;  and  (3)  to  produce  an  operational  computer  model 
which  will  incorporate  the  predicted  student  and  system  performance  data  and  can 
be  used  to  calculate  the  expected  costs  and  benefits  of  implementing  selected  CBT 
innovations. 

To  meet  the  preceding  goals,  the  following  objectives  were  specified: 

a.  To  describe  the  CBT  characteristics  expected  to  be  related  to  measure- 
able  outcomes. 

b.  To  identify  those  outcomes  that  most  reflect  "benefits"  attributable  to 
innovations,  e.g.,  student  training  time  reduction  and  decreased  elimination 
rates. 

c.  To  specify  the  relationships  and  interactions  between  all  system 
parameters . 

d.  To  identify  those  parameters  for  which  quantifiable  data  are  available. 

e.  To  determine  the  type  of  modeling  procedures  most  appropriate  for  the 
predictive  model,  e.g.,  appropriate  statistical  or  mathematical  models  for 
predicting  benefits  and  costs. 

f.  To  integrate  the  predictive  model  in  a computer  program  that  can  be 
used  as  a planning  aid. 

The  anticipated  and  actual  problems  related  to  these  goals  and  objectives 
are  discussed  and  summarized  in  the  following  section. 

1.3  LITERATURE  REVIEW 

1.3.1  Selection  of  Criterion  Variable  - Only  a few  pertinent  reports  con- 
cerning the  quantification  of  learning  system  features  in  terms  of  time-to- 
complete  a unit  of  instuction,  student  performance,  and  training  cost  were  identi- 
fied. A seminal  study  reported  by  Wagner,  Behringer,  and  Pattie  (1973)  addressed 
the  description  of  training  system  features  which  predicted  course  completion  time. 
Their  study  combined  course-related  instruments  and  standardized  aptitude  tests 
to  predict  time-to-complete.  These  devices  included: 
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0 Tests  of  skills  and  knowledge  necessary  for  the  acquisition  of  certain  j 

mi  1 ita ry  occupational  specialties  | 

0 Tests  that  sampled  the  content  of  the  tasks  to  be  learned  during  the  | 

I 

course  1 

0 Scales  designed  to  measure  motivational  factors  related  to  the  subject  ! 

matter  of  the  course. 

A step-wise  multiple  linear  regression  analysis  was  applied  to  the  data.  Correla- 
tions between  the  predictor  variables  and  course  completion  time  ranged  from  .65 
to  .87.  These  results  supported  the  hypothesis  that  course-related  predictors 
are  better  than  measures  of  general  aptitude  in  estimating  the  time- to- complete 
a course  of  instruction.  In  addition,  the  authors  accepted  the  hypothesis  that 
the  relationship  between  course-related  predictors  and  completion  time  variables 
is  linear. 

As  an  attempt  to  categorize  and  integrate  the  diverse  variables  influencing 
the  instructional  process,  Carroll  (1962,  1963)  presented  a conceptual  model  of 
school  learning  which  has  proven  to  be  quite  durable  over  the  intervening  fourteen 
years.  The  model  output,  the  criterion  variable,  is  degree  of  learning,  i.e., 
performance  on  an  achievement  test  after  a fixed  amount  of  elapsed  time.  The 
model  itself  consists  of  five  elements,  two  pertaining  to  the  characteristics 
of  the  instructional  environment,  and  three  to  the  characteristics  of  the  student. 

The  first  instructional  variable  concerns  the  adequacy  of  presentation  of  the 
learning  task,  how  clearly  it  is  presented  and  explained,  and  how  appropriately  it 
placed  in  the  sequence  of  tasks  to  be  learned.  The  second  represents  the 
opportunity  for  learning,  the  time  allowed  to  learn  the  task.  The  first  of  the 
three  individual  difference  variables  represents  the  student's  ability  to  understand 
the  instruction.  Carroll  suggests  this  to  be  analogous  to  general  or  verbal 
intelligence  and  determines  the  extent  to  which  the  student  is  able  to  understand 
directions  and  explanations  or  to  infer  them  from  the  content  of  instruction  if 
they  are  missing.  The  second,  aptitude,  is  defined  as  the  time  needed  by  the  student 
to  learn  a particular  task  to  criterion,  given  that  the  instruction  is  presented 
well  enough  for  him  to  grasp  it.  Apti tude  is  considered  to  be  a relatively  invarient 
characteristic  of  the  individual.  The  third  element  represents  the  maximum  amount 
of  time  an  individual  will  apply  himself  to  a task.  This  is  called  perseverance. 
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A recent  model  of  the  instructional  process  proposed  by  Cooley  and  Lohnes 
(Glaser,  1976)  deviates  only  ilightly  from. Carroll ' s earlier  formulation.  Carroll's 
elements  of  aptitude  and  abil  ty  to  understand  instruction  are  replaced  by  a single 
variable,  initial  ability,  representing  the  sum  of  the  student's  skills  and  intel- 
lectual capacities.  Carroll's  adequacy  of  presentation  is  replaced  by  two  elements: 
(a)  the  extent  to  which  the  curriculum  is  structured,  the  methods  of  individuali- 
zing instruction,  and  the  general  organization  of  instruction;  and  (b)  the  quality 
of  the  student-teacher  interaction.  The  term  motivation  has  been  substituted  for 
perseverance , but  it  is  defined  as  the  student's  tendency  to  engage  in  learning 
activities  when  the  opportunity  exists.  Finally,  the  relationships  among  the 
elements  are  not  stated  as  explicitly.  Rather,  these  relationships  are  to  be 
determined  empirically  through  multivariate  analysis. 

Neither  the  Carroll  nor  the  Cooley  and  Lohnes  model,  as  stated,  is  appropri- 
ate to  a self-paced,  criterion-referenced  instructional  environment.  In  both,  the 
criterion  variable  represents  degree  of  learning,  and  opportunity  for  learning 
is  assumed  to  be  limited.  In  a criterion-referenced  environment,  degree  of 
learning  is,  theoretically  at  least,  controlled  while  opportunity  for  learning 
remains  free  to  vary.  Consequently,  time  per  unit  of  instruction  (time  to 
criterion)  or  learning  rate  is  the  most  appropriate  criterion  variable  for  a model 
of  learning  in  a self-paced  environment. 

1.3.2  Selection  of  Predictor  Variables  - Psychologists  and  educators  have 
long  been  aware  that  learning  and  performance  on  instructional  tasks  are  a func- 
tion of  the  student's  level  of  ability  or  aptitude  and  his  motivation.  More 
recently,  they  have  also  recognized  the  importance  of  taking  into  account  both 
situational  and  task  factors.  The  search  for  those  variables,  however,  and 
their  associated  measures  which  best  describe  the  student's  actual  and  expected 
performance  remains  a pervasive  problem  further  complicated  by: 

0 The  difficulty  in  obtaining  quantitative  measures  of  situational  and 
task  variables  (Uprichard,  1975) 

0 The  fact  that  a student's  trait  levels  of  ability  and  motivation  may  not 
be  as  predictive  of  his  performance  as  more  si tuational ly  sensitive  state 
measures  (Speilberger , 1966;  Spielberger,  O'Neil,  and  Hansen,  1972) 
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0 The  limited  progress  that  has  been  made  in  the  identification  of  satis- 
factory measures  of  student  motivation  (Bern  and  Allen,  1974). 

Additional  problems  are  raised  when  the  concern  is  prediction  of  performance 
in  a self-paced,  individualized,  criterion-referenced  instructional  environment. 

A majority  of  research  on  variables  related  to  learning  has  been  conducted  in 
group-paced,  conventional  instructional  environments,  (Campbell  and  Schwen,  1971; 
Schwen,  1973),  and  existing  theories  and  data  on  ability  and  motivational 
variables  have  focused  on  achievement  scores  rather  than  learning  rates  as  the 
dependent  variable  of  interest  (Brown  and  James,  1972;  Colton,  1974;  McAvoy, 

Kalin,  and  Franklin,  1973). 

1.3. 2.1  Student  Abi 1 i ty/Apti tude  - A common  approach  to  the  problem  of 
identifying  student  characteristics  related  to  performance  in  a complex  instruc- 
tional system  is  some  form  of  task  analysis.  Campbell  and  Schwen  (1971)  argue 

for  an  analysis  of  learning  tasks  in  terms  of  the  effects  they  have  on  the  student's 
cognitive  processes  and  personality  variables.  They  stress  that  possible  inter- 
actions between  task  and  learner  characteristics  be  considered.  Schwen  (1973) 
suggests  that  a model  most  appropriate  for  analysis  of  learner  characteristics 
is  one  that  combines  empirical  methods,  e.g.,  predictive  statistical  models, 
with  task  analysis.  The  important  point  is  that  such  analytical  methods  can  be 
used  to  identify  potentially  important  variables  or  interactive  relationships 
between  variables  that  should  be  considered  in  a student  ability/aptitude  para- 
meter. 

1.3. 2. 2 Situational/Task  Variables  - In  considering  relevant  situational/ 
task  variables,  it  is  instructive  to  distinguish  between  those  variables  which  may 
vary  between  students,  such  as  type  of  instructional  treatments,  and  those  which 
are  constant  for  an  individual  student  but  vary  across  tasks,  e.  g. , cri terion 
levels  or  task  difficulty.  Just  as  analyses  of  learning  tasks  and  learner 
characteristics  were  recommended  for  the  identification  of  an  ability/aptitude 
parameter,  it  is  an  equally  appropriate  approach  to  the  identification  of 
relevant  situational/task  variables  (Campbell  and  Schwen,  1971;  Schwen,  1973). 
Walberg  (1975)  has  stressed  the  need  for  analyses  which  identify  other  possible 
variables  predictive  of  student  performance.  He  bases  this  on  the  fact  that, 
although  aptitudes  or  abilities  can  be  reliably  measured,  they  can  usually  account 
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for  no  more  than  half  the  variance  in  student  performance.  Uprichard  (1975) 
suggests  that  organizational  stucture  or  classroom  variables  be  taken  into  account 
as  well  as  evaluation  variables,  e.g.,  criterion  levels  and  frequency  of  tests. 

Some  additional  situational/task  variables  which  may  be  important  in  a self- 
paced  instructional  environment  are  suggested  by  the  Wagner  et  al  (1974) 
study.  Among  their  findings  were: 

0 Total  time  could  not  be  predicted  as  well  as  instructional  time  due  to 
many  unknown  factors  related  to  variabilities  in  testing  time 

0 Grouping  of  data  by  instructional  mode  was  required  in  order  to  predict 
completion  times  more  accurately. 

These  findings  suggest  that  potentially  important  situational/task  variables  would 
include  ratios  of  instructional  time  to  testing  time,  relative  difficulty  level 
of  each  instructional  unit  of  interest,  testing  frequency,  ratio  of  audio-visual 
to  printed  materials,  and  the  frequency  with  which  an  adaptive  model  choice 
(if  present  in  the  system)  constitutes  the  actual  assignment. 


ii 

li 
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2 . 0 OVERVIEW  OF  THE  TECHNICAL  APPROACH 

The  work  described  herein  consists  of  three  major  activities: 

a)  Definition  of  a descriptive  model  of  a generalized,  computer-based 
training  system. 

b)  Generation  of  a predictive  model  of  student  performance  in  a computer- 
based  training  environment. 

c)  Development  of  a computer  program  incorporating  the  predictive  per- 
formance model  to  estimate  the  cost  of  such  a system. 

The  primary  purpose  of  the  descriptive  model  was  to  identify  the  component 
activities  involved  in  the  instructional  process  in  a computer-based  training 
environment.  Further,  the  model  served  to  identify:  (a)  subject  matter  charac- 

teristics; (b)  instructional  and  system  design  considerations;  (c)  student  indi- 
vidual difference  characteristics ; and  (d)  the  relationship  of  these  variables  to 
the  various  components  of  the  model.  The  variables  to  be  considered  by  the  pre- 
dictive model  served  to  define  the  domain  of  computer-based  training  systems  to  be 
addressed  by  the  predictive  model.  As  defined,  the  model  assumed  a criterion 
referenced  testing  environment  and  emphasized,  as  the  dependent  variable,  the  time 
to  reach  criterion  on  an  arbitrary  unit  of  instruction. 

Since  this  study  focused  on  the  prediction  of  student  time-to-completion  of  a 
unit  of  instruction  (so  as  to  demonstrate  time-savings  and  cost-effectiveness), 
a model  to  account  for  prediction  of  time  was  developed.  The  analysis  of  data 
within  this  model  necessitated  the  use  of  a statistical  technique  through  which 
one  can  analyze  the  relationship  between  the  criterion  or  dependent  variable 
and  a set  of  predictor  or  independent  variables. 

The  statistical  technique  chosen  for  this  effort  was  multiple  regression. 

This  statistical  treatment  is  useful  primarily  because  of  its  ability  to  produce 
linear  prediction  equations  (Kerlinger  and  Pedhazer,  1973)  as  well  as  computing 
statistics  that  provide  the  researchers  with  statistical  inference  procedures 
for  determining  confidence  limits  for  estimates  with  a means  for  measurement  of 
variance  in  the  dependent  variable  accounted  for  by  the  predictor  set.  Finally, 
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the  u^e  of  multiple  regression  provided  the  authors  with  a degree  of  control  over 
extraneous  variables  in  order  that  the  combined  contribution  of  a specific 
set  of  variables  could  be  more  closely  examined. 

Specifically,  the  intent  was  to  develop  a model  in  the  form  of  a regression 
equation  in  which  the  criterion  was  time-to-complete  an  arbitrary  instructional 
unit.  Predicto"  variables  were  drawn  from  the  several  categories  defined  by  the 
descriptive  model  and  were  limited  to  the  type  of  information  which  will  be 
available  to  or  can  easily  be  obtained  by  a military  training  course  developer. 

The  actual  predictor  variable  data  were  sampled  from  course  planning  documents 
for  two  courses  which  had  been  group-paced  but  are  currently  being  conducted  in 
a self-paced  mode  under  the  control  of  a computer-based  training  system  - - 
the  Advanced  Instructional  System  (AIS)  at  Lowry  Air  Force  Base,  Colorado. 

The  predictor  variables  fell  into  four  broad  categories.  The  first  concerned 
course  content,  information  which  was  either  available  directly  from  the  course 
planning  documents  or  which  could  be  derived  from  these  documents.  The  second 
category  consisted  of  instructional  design  considerations.  Variables  in  this 
category  were  broadly  defined  and  limited  by  the  consideration  of  only  those 
which  varied  between  or  within  the  two  AIS  courses.  The  third  consisted  of  test 
characteristics,  the  performance  standards  established  for  the  two  courses.  The 
fourth  category  of  predictor  variables  consisted  of  student  individual  difference 
measures.  Two  classes  of  such  measures  were  considered:  those  which  would 

normally  be  available  to  the  developers  of  a military  training  course;  and  an 
additional  set  of  measures  which  was  available  to  this  study  because  of  its  use 
in  the  AIS  pre-assessment  battery. 

The  criterion  variable  explicitly  predicted  by  the  regression  equation  is 
first-attempt-time  on  an  AIS  lesson.  The  model  was  then  expanded  to  address  the 
time  required  to  complete  a larger  instructional  unit  consisting  of  lessons  and  an 
end-of-unit  criterion  test.  Lesson  and  unit  level  criterion  data  were  drawn  from 
the  AIS  student  performance  data  bases  for  the  Inventory  Management  and  Weapons 
Mechanic  courses. 

The  most  relevant  predictors  in  each  of  the  four  categories  of  predictor 
variables  were  determined  through  a series  of  linear  model  comparisons.  In  such 
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comparisons,  two  equations,  one  including  and  the  other  excluding  the  variable(s) 
of  interest  within  variable  categories,  are  evaluated  in  terms  of  the  reduction  in 
the  square  of  the  multiple  R from  the  first  to  the  second  linear  model.  In  the 
comparisons  made,  primary  emphasis  was  placed  on  the  cost  of  obtaining  data  for 
the  variable(s)  being  considered  for  exclusion  relative  to  its  (their)  predictive 
value.  Following  reduction  to  a parsimonious  set  of  predictors,  stepwise  multiple 
linear  regression  was  used  to  obtain  a final  predictive  model.  This  final  model 
was  then  cross-validated  and  tested  for  generality  as  an  additional  evaluation 
of  validity  across  courses. 

The  final  activity  in  the  study  was  the  production  of  a computer  cost  analysis 
program  entitled  Aid  for  Instructional  Development  and  Evaluation  (AIDE).  AIDE  is 
a modification  of  the  Rand  cost  analysis  program  MODCOM,  Method  of  Designing 
Instructional  Alternatives  (MODIA)  (Hess  and  Kantor,  1976)  set  of  programs.  The 
modification  consisted  of  reprogramming  MODCOM  to  use  the  predictive  model  regres- 
sion equation  developed  in  this  study  while  retaining  all  of  MODCOM's  cost  analysis 
features.  The  program  output  is  a series  of  tables  which  describe  the  course 
specifications  including  the  student  and  lesson  variable  values  employed  by  the 
predictive  model  regression  equation,  resource  requirements  and  costs,  and  the 
total  course  cost.  To  illustrate  its  use,  the  AIDE  program  was  exercised  in 
answering  three  sample  questions  related  to  student  characteristics  and  course 
design  (See  Section  5.4). 
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3.0  DESCRIPTIVE  MODEL  OF  COMPUTER-BASED  TRAINING  SYSTEMS 


Antecedent  to  the  development  of  the  computer-based  predictive  model,  it  was 
necessary  to  define  the  scope  of  the  instructional  process  to  be  modeled.  This 
was  done  through  the  generation  of  a descriptive  model.  This  descriptive  model 
provided  a basis  for  (a)  conceptualizing  the  overall  instructional  process,  (b) 
identifying  gaps  in  the  conceptualization,  (c)  identifying  variables  which  could  be 
quantified  for  inclusion  in  the  predictive  model,  (d)  determining  data  collection 
requi rements , and  (e)  assisting  in  the  interpretation  of  results.  What  is  pre- 
sented is  not  intended  to  represent  the  only  way  or  the  optimal  method  of 
accomplishing  individualized,  self-paced  instruction,  but  rather  it  is  an  attempt 
to  represent  a generalized  model  which  can  encompass  a wide  variety  of  computer- 
based  training  systems.  The  descriptive  model  was  developed  in  two  stages.  The 
first  stage,  essentially  the  context  for  the  descriptive  model  itself,  is  illustra- 
ted in  Figure  1 . 

The  most  basic  element  defining  the  characteristics  of  an  instructional 
process  model  would  appear  to  be  the  criterion  behaviors  to  which  the  student 
is  to  be  trained  - the  set  of  skills  assumed  to  be  required  in  the  field.  In 
the  Air  Force,  these  skills  are  rather  explicitly  defined  by  the  Specialty  Training 
Standard  (STS)  for  each  field. 

The  curriculum,  the  instructional  content  of  a particular  course,  represents 
the  interpretation  of  these  behavioral  goals  by  the  course  personnel.  Since  the 
curriculum  is  defined  by  the  course  personnel,  it  must  be  assumed  that  it  will  be 
shaped  by  their  cumulative  experience,  both  in  the  field  and  in  the  classroom, 
and  by  their  knowledge  of  local  institutional  constraints,  e.g.,  availability 
of  training  devices,  usual  ratio  of  instructional  personnel  to  students,  and  physi- 
cal plant  limitations.  The  final  definition  of  the  curriculum  is  represented  by 
the  course  planning  documents.  For  the  Air  Force,  this  is  the  Plan  of  Instruction 
(POI).  These  documents  contain  an  explicit  statement  of  the  course  content,  the 
amount  of  instruction  (time)  to  be  devoted  to  each  topic,  the  instructional  method 
(e.g.,  lecture  or  demonstration)  where  available,  and  the  training  aids  (e.g., 
films,  manuals  and  actual  equipment  to  be  employed).  Throughout  this  study,  it 
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win  be  assumed  that  the  course  to  be  modeled  is  in  existence,  probably  in  a 
conventional,  group-paced  format,  and  that  course  planning  documents  are  avail- 
able. 

Since  the  purpose  of  the  predictive  model  being  developed  is  to  represent  a 
self-paced,  individualized  course,  it  is  assumed  that  in  situations  in  which  the 
model  is  to  be  employed,  a complete  set  of  instructional  materials,  representing 
the  entire  course  content,  will  need  to  be  selected  or  developed.  Thus,  the  next 
block  in  the  diagram  represents  the  means  by  which  the  instructional  content  is  to 
be  presented.  These  are  the  defining  characteristics  of  the  course  of  instruction 
to  be  developed.  Is  the  instructional  method  to  be  expository  or  oriented  toward 
discovery  and  problem  solving?  Is  the  presentation  narrative  or  programmed,  that 
is,  frame  oriented  with  embedded  questions?  If  progratmied , what  is  the  frequency 
of  interaction?  To  what  extent  are  pre-  and  post-organizers,  such  as  objectives 
and  post-questions  employed?  Is  the  instruction  merely  self-paced,  or  is  it  indivi- 
dualized in  the  sense  of  advanced  placement  and  assignment  of  alternative 
instructional  treatments?  What  presentation  media  are  employed,  e.g.,  print,  audio- 
visual, or  computer-assisted  instuction?  As  is  indicated  by  Figure  1,  these 
instructional  characteristics  are  a function  of  several  considerations:  the 

content  to  be  taught;  explicit  decisions  regarding  instructional  design,  indivi- 
dualization strategies,  and  media;  the  availability  of  instructional  resources; 
and  the  influence  of  institutional  constraints  and  policies.  The  category  of 
instructional  resources  is  meant  to  include  presentation  media  (printed  materials, 
audio-visual  devices,  terminals  for  computer-assisted  instruction),  the  materials  J 

and  equipment  which  the  student  is  being  trained  to  work  with  in  the  field  (e.g., 
manuals,  actual  devices  and  tools,  simulators),  and  terminals  for  interacting  with 
a computer-management  system. 

The  instructional  interaction,  the  process  through  which  learning  takes  place, 
is  primarily  a function  of  the  instructional  content,  mode  of  presentation,  and  the  ^ 

characteristics  of  the  students  at  the  time  of  their  entry  into  the  course  - the 
level  of  their  aptitudes  and  abilities.  Although  not  shown  explicitly  in  Figure  1, 
student  characteristics  will,  hopefully,  have  been  taken  into  consideration  by 
the  instructional  design  process  and,  to  the  extent  necessary,  reflected  in  the 
course  planning  documents.  It  is  unreasonable  to  expect,  however,  that  the  instruc- 
tional presentation  will  be  perfectly  matched  to  student  characteristics.  There 


3-3 

mtCDOni^CLt.  DOUGLAS  ASTRONAUTICS  CONfRANT  - CAST 


EVALUATING 
INSTRUCTIONAL  SYSTEMS 


VOLUME  II 


MDC  El 570 
15  MAY  1977 


is  also  the  possibility  that  the  average  ability  level  of  the  students  will  vary 
over  time  as  a function  of  demand  for  trained  personnel  and  economic  fluctuations 
in  the  civilian  sector.  Finally,  the  total  distribution  of  student  charac- 
teristics, as  well  as  the  average  level,  must  be  considered  in  evaluating  the 
entire  training  system  throughout. 

It  must  be  recognized  that  the  nature  of  the  instructional  interaction  is 
also  shaped  by  existing  course  policies.  These  are,  unfortunately,  intangible 
factors  concerning  the  prevailing  attitudes  among  the  instructors  and  their  super- 
visors, and  the  incentives,  positive  and  negative,  explicit  and  implicit,  to  which 
the  students  are  exposed.  Finally,  the  instructional  interaction,  particularly  the 
time  it  requires,  is  directly  a function  of  the  actual,  day-to-day  availability 
of  the  instructional  and  management  resources.  While  resource  availability  was 
presumably  taken  into  consideration  by  the  instructional  system  design,  problems 
may  arise  if  the  student  entry  rate  is  substantially  increased  or  if  equipment 
maintenance  is  not  adequate.  In  that  case,  student  queuing  for  resources  can 
account  for  a substantial  portion  of  the  total  training  time. 

There  is  undoubtedly  a host  of  measures  by  which  the  effectiveness  of  the 
instructional  interaction,  of  the  total  instructional  system,  can  be  judged. 

The  most  tangible  measures,  however,  are  level  of  mastery,  rate  of  attrition, 
and  time-to-criterion.  As  will  be  discussed  in  subsequent  sections  of  this  report, 
level  of  mastery  is  exceedingly  difficult  to  capture  in  a model  intended  to  genera- 
lize across  training  programs.  Since  the  prime  purpose  of  the  predictive  model  is 
to  provide  a tool  for  the  comparison  of  alternative  instructional  systems,  all 
of  which  are  criterion  referenced,  it  will  be  assumed  that  the  required  level  of 
mastery  is  primarily  a policy  decision  that  remains  constant  from  one  system  to 
another.  If  level  of  mastery  is  indeed  fixed,  the  only  two  parameters  left  free 
to  vary  are  rate  of  attrition  and  time-to-criterion.  While  attrition  would  appear 
to  be  a plausible  dependent  variable,  it  has  been  the  authors'  experience  that,  in 
a miltary  training  environment  at  least,  it  is  also  largely  a function  of  course 
policy.  Thus,  the  primary  dependent  variable  to  be  considered  by  the  model  will  be 
time-to-criterion,  the  time  required  to  complete  the  course  of  instruction. 

Given  the  dependent  variable  of  time-to-completion,  the  next  step  was  to 
develop  a description  of  the  instructional  interaction  process  which  would 
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(a)  account  for  time-to-completion , (b)  generalize  across  a variety  of  computer- 
based  training  systems,  and  (c)  incorporate  the  classes  of  independent  variables 
outlined  in  Figure  1 and  the  preceeding  paragraphs.  Finally,  it  was  desirable 
that  the  model  be  sufficiently  fine  grained  that  each  of  the  individual  components 
of  the  model  could  be  related  to  a small  subset  of  the  independent  variables  and 
that  the  times  attributable  to  each  component  sum  to  equal  total  time-to-complete 
the  instructional  unit.  This  descriptive  model  is  shown  in  Figure  2. 

A course  structure  is  assumed  in  which  the  curriculum  consists  of  one  or  more 
major  instructional  units.  Each  unit  is  assumed  to  end  with  a criterion  referen- 
ced examination  which  the  student  must  pass  in  order  to  proceed  to  the  next  unit, 
or,  in  the  case  of  the  last  unit,  to  be  graduated  from  the  course.  In  the  descriptive 
model,  this  examination  has  been  termed  a credential  1 ing  test.  In  Air  Force  tech- 
nical training,  the  corresponding  instructional  unit  is  a "Block."  Student  scores 
on  end-of-block  tests  are  entered  in  the  student's  permanent  record;  a student 
who  meets  criterion  on  the  block  test  is  considered  to  have  demonstrated  proficiency 
on  the  block  content,  and  decisions  concerning  academic  elimination  from  the 
course  are  made  on  the  basis  of  block  test  scores  and  times-to-block-completion. 

It  is  assumed  that,  up  to  a point,  a student  failing  to  pass  the  credential  1 ing 
test  will  be  reassigned  part  or  all  of  the  instructional  unit  and  then  retested. 

The  descriptive  model  is  intended  to  account  for  the  time  required  for  a student 
to  complete  an  arbitrary  instructional  unit  - to  reach  criterion  on  the  unit 
credential  1 ing  test. 

Within  each  major  instructional  unit,  it  is  assumed  that  the  curriculum  is 
divided  into  a number  of  smaller  units,  each  covering  one  or  more  objectives, 
which  will  be  termed  lessons.  It  is  further  assumed  that,  in  general,  each  lesson 
has  a criterion  referenced  test  over  the  lesson  content.  Lesson  tests  are  intended 
to  be  for  diagnostic  rather  than  credential  1 ing  purposes.  That  is,  the  primary 
purpose  of  the  diagnostic  tests  is  to  track  student's  progress  through  the  course, 
to  detect  specific  areas  of  weakness  and  misunderstanding , and  to  provide  a basis 
for  lesson  level  remediation  in  these  areas.  Diagnostic  test  scores  would  not,  as 
a rule,  be  made  part  of  the  student’s  permanent  record. 

Turning  now  to  the  components  of  the  model,  the  first  decision  point. 

Determine  Set  of  Assignments,  represents  a form  of  individualization  which,  in 
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FIGURE  2 DESCRIPTIVE  MODEL  OF  TIME  TO  COMPLETE  AN  INSTRUCTIONAL  UNIT  UNOI 
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any  particular  system,  may  be  absent  or  present  and,  if  present,  may  be  present 
in  varying  degrees.  It  serves  two  functions.  On  a student's  first  pass  through 
the  instructional  unit,  it  is  analogous  to  advanced  placement  - on  the  basis  of 
known  information  about  the  student's  abilities  and  current  skills,  is  it  pro- 
bable that  he  could  skip  one  or  more  lessons  without  seriously  jeopardizing  his 
chances  of  meeting  criterion  on  the  credential  1 ing  test?  Similarly,  on  a remedial 
pass  through  the  unit  (following  failure  to  meet  criterion  on  the  credential  1 ing 
t^st),  what  minimal  set  of  lessons  (or  partial  lessons)  can  be  assigned  and  still 
have  reasonable  assurance  that  the  student  will  meet  the  credentialling  test  cri- 
terion on  a retest?  Within  the  descriptive  model,  this  component  is  represented 
by  the  probability  (p^)  that  the  student  will  be  assigned  a given  lesson  (lesson  i). 
It  is  assumed  to  be  a function  of  the  system's  capability  for  individualization, 
the  content  of  lesson  i,  the  characteristics  of  the  credentialling  test,  and  the 
student's  specific  abilities  and  skills. 

The  second  decision  point.  Make  Next  Assignment,  is  similar  in  that  it  also 
represents  a form  of  individualization  which  may  be  absent  or  present  in  varying 
degrees  in  different  instructional  systems.  It  is  intended  to  represent  system 
capability  for  individualized  assignment  of  alternative  instructional  treatments 
at  the  lesson  level.  Time  savings  attributable  to  this  component  are  represented 
as  being  distributed  across  other  components  in  the  model.  That  is,  individua- 
lized assignment  of  alternative  treatments  will  impact  both  first  attempt  ano 
remedial  study  times,  the  probability  of  passing  the  diagnostic  tests,  and,  to  an 
extent,  the  time  required  for  testing  itself. 

The  next  component.  Queuing  for  Lesson  i Resources,  is  the  first  to  represent 
a time  consuming  activity,  any  delay  due  to  the  unavailability  of  lesson  resources 
which  the  student  may  encounter  prior  to  being  able  to  start  his  next  assignment. 

It  has  lower  limit  of  zero  time  and  is  assumed  to  be  a function  of  both  the  quantity 
of  available  resources  of  the  needed  type  and  the  system's  capability  for  resource 
management,  for  scheduling  assignments  so  as  to  avoid  resource  bottlenecks. 

Tne  component  entitled  First  Attempt  Study  on  Lesson  i represents  the  amount 
of  time  consumed  by  the  student's  initial  study  of  the  lesson  i content  on  a given 
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pass  through  the  instructional  unit,  either  his  first  pass  through  the  unit  or 
unit  level  remediation.  This  component  accounts  for  the  major  portion  of  the 
student's  first  pass  time  in  the  instructional  unit  and  is  assumed  to  be  a function 
of  the  lesson  content,  system  capability  for  individualization  (the  availability 
and  individualized  assignment  of  alternative  instructional  treatments),  instruc- 
tional design  and  presentation  considerations,  the  known  (to  the  student) 
characteristics  of  the  lesson  i diagnostic  test,  and  the  characteristics  of  the 
student  himself. 

3.1  FIRST  ATTEMPT  DIAGNOSTIC  TEST  TIME. 

The  amount  of  time  which  the  student  spends  completing  the  test  following 
lesson  i,  has  a lower  limit  of  zero  (representing  the  case  in  which  there  is  no 
lesson  i diagnostic  test)  and  is  assumed  to  be  a function  of  the  test  characteris- 
tics. These  characteristics  include  whether  the  content  to  be  tested  is  primarily 
knowledge  or  involves  the  demonstration  of  performance  skills,  and  if  written, 
the  type  of  items  (completion,  multiple  choice,  or  true-false),  the  number  of 
items  (or  number  of  steps  in  a performance  task),  and  the  criterion  level.  While 
it  could  be  argued  that  the  amount  of  time  the  student  spends  on  the  test 
is  also  a function  of  his  level  of  knowledge,  and  hence  a function  of  the  same 
factors  that  influence  study  time,  the  simplifying  assumption  is  made,  for  the 
sake  of  parsimony,  that  these  factors  make  a relatively  small  contribution  to  the 
total  variance  in  testing  time.  They  are  taken  into  consideration  in  a subsequent 
component  representing  the  probability  that  the  student  will  meet  criterion  on 
the  diagnostic  test. 

3.2  QUEUING  TINE  FOR  LESSON  TEST  SCQRING/TRACKING. 

This  represents  the  amount  of  time  the  student  must  spend,  following 
completion  of  his  diagnostic  test,  until  his  test  has  been  scored  and  he  has 
received  his  next  assignment.  Placing  this  component  after  the  diagnostic  test 
time  component  implies  that  testing  itself  is  done  off-line.  The  position  of  the 
two  components  would  simply  be  reversed  to  represent  an  on-line  testing  situation. 
It  is  assumed  to  be  a function  of  the  number  of  terminals  available  for  student/ 
system  interaction,  the  speed  with  which  these  interactions  can  take  place,  and 
the  extent  to  which  assignment  scheduling  (resource  management)  can  prevent  the 
occurrence  of  waves  in  the  level  of  student  demand  for  terminals. 
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3.3  PASS/REMEDIATE  DECISION 
The  following  decision  point  represents  the  outcome  of  scoring  the  diagnostic 

test,  the  probability  that  the  student  has  met  the  test's  stated  criterion. 

If  so,  it  is  assumed  that  he  proceeds  to  his  next  lesson  assignment,  if  any.  If 
not,  it  is  assumed  that  he  is  assigned  a remedial  activity  designed  to  enable  him 
to  meet  criterion  on  the  previously  failed  objectives.  This  probability  is  assumed 
to  be  a function  of  the  content  of  lesson  i,  the  system's  capability  for  indivi- 
dualized assignment  of  alternative  instructional  treatments,  instructional  design 
considerations  for  lesson  i,  the  characteristics  of  the  lesson  i diagnostic  test, 
and  the  relevant  characteristics  of  the  student  himself.  If  there  is  no  diag- 
nostic test  corresponding  to  lesson  i,  the  decision  would,  of  course,  always  be 
"Pass." 

Let  us  first  consider  the  case  in  which  the  student  fails  to  meet  criterion 
on  the  diagnostic  test  and  is  assigned  a remedial  activity.  The  bulk  of  this 
activity  is  represented  by  the  component  labeled  Remedial  Study  on  Lesson  i. 

The  amount  of  time  required  for  this  activity  is  assumed  to  be  a function  of  the 
same  factors  which  determined  first  attempt  study  time  on  the  lesson  and,  in 
addition,  course  training  policy  considerations.  The  intent  of  including  this 
policy  factor  is  to  allow  for  situations  in  which  the  local  training  policy 
dictates  that  the  student  retest  within  a certain  period  of  time  or  that  the 
student  spend  at  least  a minimum  period  of  time  before  retesting. 

It  will  be  noted  that  the  component  Queuing  for  Lesson  i Resources  has  not 

been  included  in  the  remedial  loop.  It  is  assumed  that  resource  management  will 

be  such  that  if  a training  device  was  required  for  the  student's  first  attempt  on 

the  lesson,  the  device  will  remain  available  to  him  for  remediation. 

< " 

3.4  REPEATED  DIAGNOSTIC  TEST  ON  LESSON  i AND  QUEUING  FOR  LESSON  TEST  SCORING/ 
TRACKING 

The  next  two  components,  are  essentially  the  same,  and  a function  of  the  same 
factors,  as  the  comparable  components  in  the  loop  respresenting  initial  study. 

It  should  be  noted  that  the  number  of  items  on  the  diagnostic  test,  and  hence  the 
time  required  for  testing,  could  change  if  a scheme  is  employed  by  which  a student 
is  retested  only  over  previously  failed  objectives. 
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The  following  decision  point,  the  probability  that  the  student  will  now 
meet  criterion,  is  analogous  to  the  prior,  first  pass  decision  point  and  is 
assumed  to  be  a function  of  the  same  factors  with  the  addition  of  a course  policy 
variable.  The  introduction  of  the  policy  factor  is  to  allow  for  those  situations 
in  which  local  policy  might  require  that  a student  oroceed  to  the  next  lesson 
assignment  after  two  or  three  lesson  attempts.  This  might  be  accomplished,  for 
example,  through  intensive  remediation  by  an  instructor  and  his  certification  that 
the  student  has  met  the  diagnostic  test  criterion.  If  the  decision  is  that  the 
student  again  failed  to  meet  criterion,  it  is  assumed  that  he  will  recycle  through 
the  remedial  loop.  A decision  that  the  student  has  now  met  criterion  returns  him 
to  the  same  point  as  was  the  case  for  a student  who  met  criterion  following  his 
initial  attempt  on  lesson  i. 


3.5  INSTRUCTIONAL  UNIT  LEVEL  COMPONENTS 

If  the  set  of  lessons  assigned  to  this  student  for  the  current  instructional 
unit  has  not  yet  been  completed  (All  Assignments  Completed?),  he  repeats  the  cycle 
described  in  the  preceding  paragraphs.  If  all  assigned  lessons  have  been  completed, 
that  is,  if  the  student  has  studied  all  assigned  content  in  the  instructional  unit, 
he  is  assigned  the  unit  credential  ling  test. 

The  component  shown  preceding  the  credential  1 ing  test.  Cumulative  Administrative 
Overhead , is  intended  to  account  for  that  time  which,  while  the  student  was  in  the 
current  instructional  unit,  was  spent  on  non-instructional  activities.  Examples 
would  be  roll  call,  extra  group  instruction  such  as  safety  lectures,  classroom 
clean  up,  assignment  to  special  details,  smoking  and  coffee  breaks-  This  time 
is  to  be  strictly  a function  of  local  course  policy  and  would  have  to  be  estimated 
in  each  specific  situation. 

Time  spent  completing  the  credential  1 ing  test  is  assumed  to  be  a function  of 
the  same  test  characteristics  as  was  the  case  for  the  diagnostic  tests.  This 
component  would,  however,  also  include  any  time  spent  in  critiquing  the  student's 
item-by-item  performance  on  the  credential! ing  test,  an  activity  assumed  to  be 
unlikely  following  the  more  frequent  diagnostic  tests. 
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Again,  allowance  must  be  made  for  the  time  required  to  complete  test  scoring. 
This  is  represented  by  the  component  labeled  Queuing  for  Unit  Test  Scoring/Tracking 
and,  as  was  the  case  for  diagnostic  test  queuing,  is  assumed  to  be  a function  of 
terminal  resource  availability. 

3.6  PASS/REMEDIATE/WASHOUT  DECISION 

The  final  decision  point  represents  the  outcome  of  the  evaluation  of  the 
student's  performance  on  the  credential  1 ing  test.  If  t' “ student  is  determined 
to  have  met  the  stated  criteria,  he  proceeds  to  the  next  instructional  unit  or  is 
considered  to  have  completed  the  course.  It  is  assumed  that  a student  who  fails 
to  meet  criterion  will  be  "washed  back"  to  repeat  some  or  all  of  the  lessons  in 
the  instructional  unit.  For  the  sake  of  parsimony,  it  is  assumed  that  this  major, 
unit-wide  remediation  can  be  modeled  in  the  same  manner  as  was  the  student's 
first  pass  through  the  unit.  The  amount  of  material  assigned  for  remediation 
may  differ  from  the  student's  first  pass  through  the  unit  depending  on  the  system's 
capability  for  individualization.  Finally,  it  is  assumed  that  the  decision  to 
eliminate  a student  from  the  course  on  the  basis  of  academic  deficiencies  will 
only  be  made  following  a credentialling  test. 

I 

1 


} 

1 
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4 . 0  PREDICTIVE  MODEL  OF  COMPUTER-BASED  TRAINING  SYSTEMS 

4.1  DESCRIPTION  OF  DATA  SOURCE 

4.1.1  Selection  Criteria  - The  criteria  established  for  selecting  a data  source 
were  as  follows: 

0 A computer-based,  self-paced,  individualized  instructional  system  well 
into  its  development  cycle 

0 The  availability  of  quantitative  and  qualitative  course  content  descriptions 
from  both  the  prior,  conventional  mode  of  instruction  and  from  the  current 
self-paced  mode 

0 The  availability  of  pre-instructional  measures  of  trainee  characteristics 
0 The  availability  of  trainee  performance  (time  and  score)  data  under  the 
computer-based  system. 

Of  the  available  sources,  the  Air  Force  Advanced  Instructional  System  (AIS) 
at  Lowry  Air  Force  Base,  Colorado,  was  considered  the  best  to  meet  these  criteria. 

In  addition,  resident  specialists,  familiar  with  the  characteristics  and  use  of 
the  AIS  data  base  were  available  as  consultants.  Computer  time  and  a limited 
amount  of  software  support  were  also  made  available. 

4.1.2  The  Advanced  Instructional  System  - AIS  is  being  developed  by  the  McDonnell- 
Douglas  Corporation  under  a contract  with  the  Air  Force  Human  Resources  Laboratory 
(TT),  Lowry  Air  Force  Base  (Rockway  and  Yasutake,  1974;  Yasutake  and  Stobie,  1975). 
It  is  intended  to  be  a comprehensive  CMI/CAI  system  for  the  administration  and 
management  of  large  scale,  individualized  technical  training.  At  the  time  of  this 
study,  four  courses,  containing  approximately  600  students,  were  being  supported 
with  the  overall  goal  of  achieving  an  average  25  percent  reduction  in  training  time 
over  what  was  required  prior  to  system  implementation.  These  four  courses  represent 
a wide  range  of  learning  requirements,  from  clerical  to  mechanical  to  problem 
solving  skills,  and  cover  a variety  of  course  administrative  requirements.  In 
addition,  AIS  serves  as  a research  and  development  tool  to  allow  evaluation  of  a 
variety  of  innovations  in  instructional  technology,  particularly  those  aimed  at  the 
adaption  of  instruction  to  student  individual  differences. 

A large,  central  computer  (CDC  Cyber  73-16)  provides  data  processing  capa- 
bility. Data  input  and  output  are  supported  by  two  types  of  terminals.  A 
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management  terminal  (consisting  of  a form  reader,  printer,  and  minicomputer 
enclosed  in  a console)  is  used  to  read  and  score  test  forms,  transmit  data  to  the 
central  computer,  and  print  out  student  prescriptions  and  management  reports. 

An  interactive  terminal  (consisting  of  a plasma  display  and  keyboard)  is  used  for 
instructor  and  course  author  interaction  with  AIS  data  files  and  will  also  be  used 
for  on-line,  adaptive  testing  and  CAI. 

The  system  was  designed  to  provide  the  functions  of  diagnosis  of  student 
performance,  forward-going  assignment  prescriptions , resource  allocation  and 
scheduling,  guidance  and  counseling,  information  retrieval,  report  generation, 
and  support  for  evaluation  and  research.  The  software  to  support  these  functions 
comprises  the  Adaptive  Model  and  Applications  Programs  component  of  the  AIS. 

This  software  was  tested  and  implemented  in  all  four  courses  during  the  summer  of 
1976. 

The  Adaptive  Model  contains  the  component  entitled  Adapter  which  assesses 
cognitive  and  affective  student  traits  and  states,  and  selects  instructional 
alternatives  that  best  match  the  needs  of  each  student  at  each  node  of  the  instruc- 
tional sequence.  The  individualization  afforded  by  the  Adapter  is  constrained  by 
the  prerequisite  relationships  inherent  in  a course  structure  and  by  the  availa- 
bility of  instructional  resources.  Thus,  a Resource  A1 location/Schedul inq  Model 
within  the  adaptive  model  maintains  the  inventory  of  available  resources,  restricts 
the  assignment  of  alternatives  to  the  available  resources,  supports  the  scheduling 
of  team  tasks,  and  attempts  to  optimize  the  utilization  of  expensive  and  limited 
training  resources. 

Both  the  Adapter  Model  and  Resource  Allocation/Schedul ing  Model  operate  on 
predictions  as  to  the  amount  of  time  each  student  will  require  to  complete  his 
current  assignment  and  his  current  course  unit.  These  data,  in  conjunction  with 
predictions  of  student  scores  on  each  criterion  test,  are  used  to  provide  the 
guidance  and  counseling  function  in  that  each  student's  performance  is  continually 
evaluated  relative  to  other  students. 


At  the  time  this  study  was  being  conducted,  the  AIS  was  operating  in  a more 

rudimentary  mode,  termed  Initial  CMI.  Although  the  courses  were  self-paced,  the 

primary  means  of  instruction  was  limited  to  programmed  textbooks.  Instructional 

alternatives  were  under  development,  but  only  a very  few  had  been  integrated  into 
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the  courses.  While  the  main  track  of  instruction  did  include  some  instances  of 
audio-visual  lessons,  the  number  and  variety  of  these  lessons  were  insufficient 
for  the  purposes  of  this  study.  While  the  system  provided  test  scoring  and  data 
recording  capabilities,  the  sequence  in  which  the  lessons  were  studied  and  the 
assignment  of  instructional  resources  were  under  the  control  of  the  classroom 
instructors.  Each  student  recorded  the  amount  of  time  he  spent  in  various 
activities,  e.g.,  first-attempt-lesson-study,  remedial  study,  and  testing, 
on  his  individual  test  answer  sheets.  These  data  were  recorded  by  the  system, 
but  there  was  no  direct  measurement  of  these  times. 

4.1.3  Courses  Selected  - The  four  courses  supported  by  AIS  at  the  time  of  the 
study  were  Inventory  Management  Specialist,  Materiel  Facilities  Specialist, 

Precision  Measuring  Equipment  Specialist,  and  Weapons  Mechanic.  The  analysis  to 
oe  conducted  required  a large  data  base  (hence  a relatively  large  student 
flow)  and  a large  overlap  between  the  content  of  the  current,  self-paced  course  and 
the  earlier,  conventional  course.  On  the  basis  of  these  considerations,  the  two 
courses  selected  for  modeling  were  Inventory  Management  Specialist  (IM)  and  Weapons 
Mechanic  (WM). 

Under  conventional  instruction,  IM  was  a 210  hour  course  leading  to  the 
Air  Force  Specialty  of  Inventory  Clerk.  Course  content  primarily  concerns 
inventory  procedures,  record  keeping,  and  shipping.  The  work  requires  clerical  and 
procedure-following  skills.  The  selection  criterion  for  admission  is  a percentile 
score  of  60  on  either  the  General  or  Administrative  Scale  of  the  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB)  given  all  enlisted  personnel  at  the  time  of 
i nduction. 

Under  conventional  instruction,  WM  was  a 540  hour  course  leading  to  a 
specialty  in  weapons  handling.  The  course  teaches  the  cognitive  and  psychomotor 
skills  required  to  handle  and  load  air-to-air  and  air-to-ground  weapons  and  to 
troubleshoot  and  repair  related  equipment.  A key  feature  of  the  WM  course  is  its 
employment  of  team  concepts  in  skill  development.  The  admission  criterion  is  a 
percentile  score  of  at  least  60  on  either  the  Electrical  or  Mechanical  scales  of 
the  ASVAB. 

Both  courses  were  divided  into  a number  of  "blocks"  (six  in  IM;  12  in  WM), 
each  approximately  a week  in  length.  Each  block  contained  from  10  to  30  lessons  and 
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an  end-of-block  test.  Students  were  required  to  meet  criterion  on  these  block 
tests  before  proceeding  to  the  next  block.  Under  AIS,  lessons  are  approximately 
one  to  two  hours  in  length;  each  covers  two  or  more  instructional  objectives, 
and  most  conclude  with  a diagnostic  lesson  test.  Under  conventional  instruction, 
the  lessons  had  tended  to  be  longer,  and,  as  a rule,  a conventional  lesson  was 
broken  up  into  two  or  more  AIS  lessons. 

Of  the  six  blocks  in  the  IM  course,  the  first  five  were  analyzed  for  this 
study.  Insufficient  student  performance  data  were  available  from  the  sixth  since 
it  had  recently  undergone  a major  revision.  The  last  four  of  12  blocks  were 
analyzed  in  the  WM  course.  The  course  was  being  implemented  under  AIS  from  back 
to  front,  and  insufficient  data  were  available  from  the  lower  numbered  blocks. 

4.2  DEFINITION  OF  VARIABLES  FOR  ANALYSIS  AND  DATA  COLLECTION  PROCEDURES 

4.2.1  Definition  of  Criterion  Variable  - Several  types  of  time  data  available 
from  AIS  were  considered  for  use  as  the  criterion  variable  to  be  predicted  by  the 
model.  These  included  block  completion  time,  first  pass  block  time,  lesson 
completion  time,  first  attempt  lesson  time,  first  attempt  study  time,  and  first 
attempt  test  time.  Of  all  of  these  time  criteria,  block  completion  time  was 
considered  to  be  the  most  useful  criterion  and,  because  of  its  relationship  to 
the  descriptive  model,  the  most  appropriate.  It  was,  therefore,  selected  as  the 
criterion  variable  to  be  predicted  by  the  model. 

For  purposes  of  developing  the  predictive  model,  however,  the  number  of  blocks 
was  too  small  to  provide  sufficient  variability  in  content.  Time  prediction 
equations  needed  to  be  developed  at  the  lesson  level.  As  was  noted  in  the  dis- 
cussion of  the  descriptive  model,  it  could  not  be  assumed  that  the  sum  of  lesson 
completion  times  would  equal  block  completion  time  or  even  first-pass  block  time. 
Each  pass  through  the  block  also  includes  the  time  required  for  the  block  test 
itself,  cumulative  administrative  overhead  which  included  student  "breaks," 
and  time  lost  due  to  queuing  for  resources  and  test  scoring.  No  accurate  time 
data  were  available  for  any  of  these  components.  The  most  appropriate  approach, 
therefore,  appeared  to  be  one  in  which  the  variable  predicted  would  be  lesson 
completion  time.  A proportional  "lost  time"  estimate,  determined  from  the  dis- 
crepancy between  the  sum  of  lesson  completion  times  and  block  completion  time, 
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and  assumed  to  contain  the  time  allocated  to  block  remediation,  overhead,  and 
queuing  would  then  be  employed  to  provide  an  estimate  of  block  completion  time. 

It  was  found,  however,  that  lesson  completion  time  was  itself  extremely 
unreliable.  The  time  required  for  each  attempt  on  the  lesson  was  recorded  on  a 
separate  test  answer  sheet  with  an  identifying  attempt  number.  Because  of  the 
multiple  pieces  of  paper  involved  and  because  procedures  were  not  always  rigidly 
adhered  to  in  the  classrooms,  the  problems  of  incomplete  and  missing  data  were 
widespread.  The  amount  of  available  lesson  completion  time  data  which  could  be 
considered  reliable  was  inadequate  for  the  modeling  study. 

Some  consideration  was  given  to  the  separate  prediction  of  reported  first- 
attempt-study  and  first-attempt-test  time.  This  approach  was  discarded,  however, 
since  there  had  been  a continual  problem  in  instructing  students  in  how  to  differen- 
tially record  study  time  and  test  time  due  to  confusion  over  self-tests  which  were 
to  be  counted  as  part  of  study  time.  The  variable  finally  selected  to  serve  as 
the  criterion  to  be  predicted  was  f i rs t-at temp t- 1 es son- time . This  variable  was 
based  on  a simple  start  and  stop  time  recorded  on  the  student's  answer  sheet  and 
had  been  found  to  be  the  most  reliable  of  the  various  time  measures.  For  the 
majority  of  cases,  it  was  equivalent  to  lesson  completion  time.  It  could  be 
related  to  block  time  by  increasing  the  magnitude  of  the  proportion  of  "lost 
time."  Viewed  in  the  context  of  the  Descriptive  Model  (Figure  2),  the  criterion 
measure  was  the  cumulative  time  (across  lessons)  attributable  to  the  two  components 
entitled  First  Attempt  Study  on  Lesson  i and  First  Attempt  Diagnostic  Test  on 
Lesson  i. 

The  prediction  of  block  and  lesson  test  scores  presented  a very  different 
problem  - that  of  general izabi 1 ity.  It  was  intended  that  the  predictive  model 
to  be  developed  should  generalize  across  a variety  of  technical  training  courses. 
While  there  is  no  ambiguity  concerning  the  meaning  of  a clock  hour  and  no  vari- 
ation in  the  meaning  of  this  concept  from  one  course  to  another,  this  is  not  the 
case  with  regard  to  test  scores.  The  probability  that  a given  student  will  obtain 
a passing  score  on  an  arbitrary  test  is  at  least  as  much  a function  of  test  item 
difficulty  as  of  the  student's  characteristics  and  the  content  being  tested. 

Item  difficulty  is,  in  turn,  primarily  a function  of  the  experience  and  opinions 
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of  the  developers  of  the  specific  instructional  materials  and  the  test.  It  is 
a safe  assumption  that  the  foremost  factor  influencing  a test  developer's  decisions 
regarding  item  difficulty  and  adequate  criterion  performance  is  his  past  experience 
with  the  performance  of  students  in  this  particular  course  of  instruction. 

Thus,  while  it  appeared  implausible  to  generalize  across  courses  with  respect 
to  expected  test  score  or  expected  failure  rates,  it  did  appear  reasonable  to  expect 
that  the  level  of  student  performance  would  remain  relatively  constant  from  con- 
ventional to  self-paced,  individualized  instruction.  This  assumption  was  made 
explicit  in  the  predictive  model  being  developed.  That  is,  it  was  assumed  that 
the  level  of  within-course  student  performance,  the  skill  level  of  the  course 
graduates,  and  the  rate  of  attrition  would  remain  unchanged  following  the  transi- 
tion from  conventional  to  self-paced,  individualized  instruction.  The  only 
dimension  left  free  to  vary  was,  therefore,  time. 

4.2.2  Definition  of  Predictor  Variables  - The  variables  selected  for  investi- 
gation as  potential  predictors  of  first-attempt-lesson  time  will  be  discussed  in 
terms  of  the  categories  of  independent  variables  influencing  the  instructional 
interaction  which  were  outlined  in  Figure  2 and  discussed  in  Section  3.0. 

While  it  would  have  been  plausible  to  consider  behavioral  goals  (see  Figure 
1 ) as  a category  of  predictor  variables,  it  was  considered  more  appropriate  to 
obtain  measures  of  the  instructional  content  itself,  the  course  curriculum. 
Information  regarding  both  classes  of  factors  were  available  - the  Specialty 
Training  Standard  (STS)  reflecting  goals  and  the  Plan  of  Instruction  (POI)  ref- 
lecting content.  While  the  information  provided  by  the  STS  is  more  basic  and 
less  subject  to  change,  the  relationship  between  the  two  documents  should  remain 
constant  unless  the  course  goals  and  content  are  subjected  to  the  type  of  re- 
analysis called  for  by  an  Instructional  System  Development  (ISD)  exercise 
(U.S.  Arniy  Training  and  Doctrine  Command,  1975).  The  purpose  of  the  predictive 
model  is  to  examine  the  effects  of  self-pacing  and  indi vidual ization,  not  the 
effects  of  ISD.  It  was  therefore  assumed  that  content,  as  reflected  in  the  POI, 
would  remain  constant  during  the  transistion  from  conventional  to  individualized 
instruction.  Use  of  the  POI  as  the  reference  document  for  course  content  has  the 
further  advantage  of  reflecting  the  influence  of  local  training  policy.  The  specific 
information  derived  from  the  POI  is  described  in  paragraph  4. 2. 2.1. 
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As  we  discussed  in  paragraph  4.1.2,  the  Initial  CMI  mode  of  operation,  which 
was  in  effect  at  the  time  of  this  study,  did  not  include  computer  controlled 
individualization  of  the  type  required  by  the  descriptive  nwdel,  i.e.,  advanced 
placement  and  assignment  of  alternative  instructional  treatments.  Therefore, 
no  variables  pertaining  to  this  factor  could  be  included  in  the  predictor  set. 

In  the  category  of  instructional  design  considerations,  it  was  not  feasible 
to  include  alternative  media  characteristics  as  predictors  because  of  the  limited 
number  of  lessons  employing  media  other  than  text.  The  extent  to  which  advance, 
embedded,  and  post-instructional  organizers  were  present  in  various  lessons  did 
provide  a basis  for  defining  a number  of  predictors.  These  are  discussed  in 
paragraph  4, 2. 2. 2.  Predictors  in  the  test  characteristics  category  are  discus- 
sed in  paragraph  4. 2. 2. 3.  In  addition,  a variety  of  measures  of  pre-instruction, 
individual  student  difference  characteri sti "“s  was  investigated.  These  are 
described  in  paragraph  4. 2. 2. 4. 

While  the  quantity  and  quality  of  instructional  and  management  system  resources 
have  a definite  effect  on  the  characteristics  of  the  instructional  interaction, 
particularly  the  time  required  for  its  completion,  there  was  not  sufficient 
variability  in  this  factor  among  the  various  AIS  courses  to  permit  its  use  as  a 
predictor  variable.  Consequently,  the  predictive  model  assumes  an  "adequate" 
supply  of  resources. 

It  was  recognized  that  course  policy  regarding  training  has  a pervasive 
influence  on  the  instructional  interaction.  It  is  by  nature,  however,  course- 
specific  and  relatively  intangible.  For  this  initial  attempt  at  model  development, 
therefore,  it  was  considered  necessary  to  ignore  this  factor  - to  consider  its 
effect  as  contributing  to  unaccounted  for  variability  except  in-so-far  as  course 
policy  influences  were  reflected  in  the  instructional  content. 

4.2.2. 1 Course  Content  Variables  - Course  content  characteristics  were 
determined  from  descriptions  contained  in  the  plans  of  instruction  (POI)  pertaining 
to  the  five  IM  and  four  WM  blocks  to  be  analyzed.  Each  of  the  Armed  Services  has 
planning  documents  containing  detailed  descriptions  of  lesson  objectives  and  the 
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distribution  of  lesson  time  into  categories  such  as  lecture,  performance,  testing, 
and  outside  assignment.  The  variables  selected  to  describe  each  lesson  were  based 
on  information  common  to  these  planning  documents. 

The  content  of  each  AIS  lesson  was  identified  ii.  1.'.^  r’OI  on  the  basis  of 
common  objectives.  In  some  cases,  the  set  of  objectives  covered  by  a single 
conventional  lesson  had  been  transferred  directly  to  an  AIS  lesson.  In  general, 
however,  the  conventional  lessons  were  longer  than  AIS  lessons  and  contained  more 
material.  In  these  cases,  two  or  more  artificial  conventional  lessons  were  defined, 
each  containing  the  objectives  corresponding  to  a single  AIS  lesson.  The  total 
instructional  time  in  the  conventional  lesson  and  the  time  allocated  to  each  of 
the  instructional  methods  by  the  POI  were  then  distributed  across  the  new,  arti- 
ficial lessons  on  the  basis  of  the  objectives  contained  in  each  new  lesson.  In  a 
similar  fashion,  if  an  AIS  lesson  contained  the  objectives  of  two  or  more  con- 
ventional lessons,  the  times  within  the  POI  categories  were  summed  to  create  an 
artificial  conventional  lesson.  If  the  depth  of  content  in  the  conventional 
lesson  had  been  reduced  or  increased  in  the  corresponding  AIS  lesson,  the  POI 
times  were  increased  or  reduced,  by  category,  by  the  corresponding  amount.  Finally, 
there  were  some  cases  in  which  course  content  had  changed  during  the  AIS  implemen- 
tation period,  and  no  conventional  instruction  POIs  were  available  which  included 
the  new  objectives.  When  this  occurred,  the  AIS  lesson  containing  the  new  objec- 
tives was  dropped  from  the  analysis. 

These  decisions  and  translations  were  made  by  members  of  the  AIS  materials 
development  teams  and  content  matter  specialists  who,  in  many  cases,  had  previously 
taught  in  the  conventional  courses.  The  conclusion  of  this  exercise  defined  a 
set  of  AIS  lessons  in  the  nine  blocks  which  could  be  characterized  by  descriptive 
statements  in  actual  POIs  or  in  POIs  constructed  artificially. 

It  was  obvious  that  one  of  the  most  promising  predictors  cf  first-attempt- 
lesson  time  under  a self-paced  system  would  be  the  amount  of  time  devoted  to  that 
lesson  under  conventional  instruction.  This  var  ble  was  defined  as  total  POI 
time.  There  was  reason  to  expect,  however,  that  predicted  first-attempt-lesson 
times  (hereafter  referred  to  simply  as  lesson  time)  would  be  more  accurate  if 
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separate  variables  were  defined  for  the  various  categories  of  instruction  outlined 
by  the  POI.  Four  such  variables  were  defined:  Lecture/Discussion ; Demonstration/ 

Performance;  Programmed  Instruction/Audio- Visual ; and.  Outside  Assignment. 

Each  AIS  lesson  contained  only  material  which  pertained  directly  to  the  stated 
objectives.  The  "nice  to  know"  information  which  would  usually  be  included  in  a 
lecture  had  been  eliminated.  Therefore,  it  was  assumed  that  the  amount  of  time 
which  the  POI  allocated  to  lecture  and  discussion  activities  would  be  shortened 
dramatically  under  self-pacing. 

Under  conventional  instruction.  Demons tration/Performance  activities  usually 
involve  at  least  two  repetitions  of  the  task  to  be  performed  - once  by  the  instruc- 
tor and  once  by  the  student.  Under  self-pacing,  the  task  was  performed  only  once 
by  the  student  under  step-by-step  guidance  by  the  instructional  materials.  This 
might  be  slower  or  faster  than  was  the  case  under  conventional  instruction. 

Analysis  of  the  objectives,  taught  at  least  in  part  through  Demonstration/ 
Performance  activities,  suggested  that  a further  breakdown  might  be  profitable. 
While  some  of  the  objectives  were  primarily  psychomotor  in  nature,  others  appeared 
to  be  primarily  cognitive  in  nature  (e.g.,  troubleshooting).  Therefore,  two  sub- 
categories were  defined:  Demons trati on/ Pe rf ormance- Cogni ti ve ; and  Demonstration/ 

Per forma nee- Psychomotor.  While  the  distinction  could  have  been  made  on  a judge- 
mental basis,  an  alternative  approach  was  employed.  Each  AIS  lesson  could,  poten- 
tially, contain  a performance  check,  supervised  by  an  instructor,  as  well  as  a 
multiple-choice  test.  If  the  AIS  materials  developers  had  seen  fit  to  include  a 
performance  check  in  the  lesson,  the  POI  Lecture/Discussion  time  was  categorized 
as  Psychomotor.  Otherwise,  the  time  was  categorized  as  Cognitive. 

POI  instructional  time  allocated  to  programmed  instruction  or  an  audio- 
visual presentation  was  grouped  under  a single  variable  on  the  assumption  that  the 
time  required  for  such  activities  under  self-pacing  would  not  be  shortened  sub- 
stantially. Finally,  Outside  Assignment  time  was  treated  separately  on  the  assum- 
ption that  such  activities  would  be  less  efficient  than  if  they  were  conducted 
in  the  classroom. 
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In  summary,  six  quantitative  measures  of  content  were  defined:  The  number 

of  minutes  allocated  by  the  POI  to  (a)  Lecture/Discussion,  (b)  Demonstration/ 
Performance-Coqniti ve,  (c)  Demonstration  Per forma nee- Psychomotor,  (b)  Programmed 
Instruction/Audio-Visual , (e)  Outside  Assignment;  and  (f)  the  total  number  of  minutes 
allocated  to  the  conventional  lesson,  the  sum  of  the  first  five  measures. 

Defining  measures  of  the  qualitative  characteristics  of  the  content  presented 
a substantially  different  problem.  While  several  investigators  have  offered  their 
views  regarding  the  interaction  of  specific  content  characteristics  and  student 
performance,  no  one  to  date  has  been  able  to  measure  the  relationship  between 
these  content  characteristics  and  rate  of  learning  in  an  operational  environment. 

The  current  study  employed  two  distinct  approaches  to  the  problem  of  content 
classification  and  conducted  a comparative  evaluation  of  the  two.  The  results  of 
this  evaluation  are  discussed  in  paragraph  4. 3. 3.1. 

The  first  approach,  which  will  be  termed  content  classification,  concentrated 
directly  on  the  lesson  content  as  described  by  the  POI.  Three  broad  categories 
were  defined:  memory,  cognitive,  and  psychomotor.  While  few  attempts  have  sought 

to  identify  such  broad  categories  (most  have  dealt  with  more  specific  factors  in 
well  defined  environments,  e.g.,  Guilford,  (1971),  it  was  thought  that  this  approach 
would  have  greater  utility  to  instructional  personnel  given  the  task  of  charac- 
terizing course  content. 

There  was  a dearth  of  personnel  qualified  and  available  to  evaluate  each 
lesson  in  terms  of  the  percent  (later  translated  into  minutes)  of  memory,  cognitive, 
and  psychometer  content  involved.  For  the  IM  course,  two  instructors  and  a third 
party,  not  a subject  matter  expert,  were  asked  to  make  these  ratings  jointly  on  the 
29  lessons  of  interest.  Only  one  subject  matter  expert  was  found  to  make  these 
evaluations  for  the  35  lessons  of  interest  in  the  WM  course.  For  each  content 
area  represented  in  these  lessons,  the  raters  were  asked  to  judge  on  a scale  of 
one  to  five  how  simple  or  complex  the  material  appeared  to  be  for  the  trainee  popu- 
lation to  whom  it  was  presented.  For  example,  the  content  of  a particular  two  hour 
lesson  might  be  judged  to  be  15  percent  (18  minutes)  memory  and  85  percent  (102 
minutes)  psychomotor.  The  memory  content  might  be  judged  as  being  difficult  and 
given  a complexity  rating  of  five.  The  psychomotor  component  of  the  lesson  might 
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be  judged  to  be  of  medium  complexity  and  given  a rating  of  three.  Since  the  lesson 
was  devoid  of  cognitive  material,  no  rating  was  required  for  that  category.  The 
precedure  of  obtaining  content  ratings  for  the  IM  and  WM  lessons,  either  by  inter- 
judge agreement  or  by  a single  judge,  respectively,  made  the  determination  of 
inter-judge  reliability  estimates  inappropriate. 

Definitions  and  examples  of  the  three  categories  and  of  the  complexity  scales 
within  each  category  are  provided  in  Table  1.  These  same  definitions  were  given 
to  the  raters  to  guide  their  review  of  the  POIs.  These  ratings  served  as  input  to 
the  data  base. 

The  second  approach  to  categorizing  lesson  content  adopted  a more  conventional 
method,  more  oriented  toward  the  terminal  behavior  described  by  the  objectives 
stated  in  the  POI.  A modification  of  Merrill's  (1972)  classification  scheme  was 
adopted.  Merrill's  categories  of  discriminated  recall,  classification,  >"ule  using, 
and  higher  order  rule  using  were  supplemented  with  the  category  of  Dsvchomotor 
tasks.  A binary  easy/difficult  dimension  was  then  added  to  each  category.  Defini- 
tions of  the  five  categories  and  the  bases  for  the  easy/difficult  decisions  are 
shown  in  Table  2. 

Two  psychologists  on  the  AIS  project  staff  classified  each  objective  listed  in 
the  relevant  portions  of  the  POIs  into  one  of  these  ten  categories.  Each  lesson 
was  then  rated  on  the  basis  of  the  proportion  of  the  content  attributable  to  the 
objective(s)  in  each  category.  For  example,  a particular  lesson  might  contain 
four  specific  performance  objectives.  Three  of  the  objectives  might  be  classified 
as  Discriminated  Recall.  Two  of  these  might  be  judged  as  Discriminated  Recall  - 
Easy  and  the  third.  Discriminated  Recall  - Difficult.  The  fourth  objective  might 
be  categorized  as  an  easy  classification  task.  The  POI  time  allocated  to  the  two 
easy  discriminated  recall  objectives  would  be  assigned  to  the  variable.  Pi scrimi nated 
Recall  - Easy,  for  this  lesson.  Ihe  time  attributable  to  the  other  two  objectives 
would  be  the  values  of  the  variables.  Discriminated  Recall  - Difficult  and  Cl ass- 
ifi cation  - Easy.  The  variables  representing  the  seven  remaining  categories  of 
terminal  behavior  or  performance,  Classification-Difficult  through  Psychomotor- 
Difficult  would  be  given  values  of  zero. 
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TABLE  1 

CONTENT  CLASSIFICATION:  DEFINITION 

AND  METHOD  OF  CALCULATION 


CONTENT  AREA 


DESCRIPTION 


Memory  Related  Content 


Memory  - Simple/Complex 


Cognitive  Content 


Cognitive  - Simple/Complex 


Psychomotor  Content 


Psychomotor  - Simple/Complex 


That  portion  of  the  lesson  content  re- 
quiring retention  of  information. 

Measured  in  terms  of  the  minutes  of 
conventional  instruction. 

A five  point  rating  of  the  simplicity 
or  complexity  of  the  memory  related 
element  in  the  lesson.  Simple  memory 
content  (one)  includes  recognition  of 
familiar  terms;  complex  memory  content 
(five)  includes  recall  of  many  new 
technical  terms. 

That  portion  of  the  content  related  to 
knowledge  gathering,  understanding 
relationships  and  principles,  and 
problem  solving.  Measured  in  terms  of 
minutes  of  conventional  instruction. 

A five  point  rating  of  the  simplicity 
or  complexity  of  the  cognitive  element 
in  the  lesson.  Simple  cognitive 
content  (one)  involves  procedure 
following,  labeling,  and  information 
collecting.  Complex  cognitive  content 
(five)  involves  symbol  manipulation, 
problem  solving,  decision  making,  and 
eval uation. 

That  portion  of  the  content  related  to 
performance  or  "hands-on"  tasks  and 
involving  manual  dexterity,  eye-hand 
coordination,  or  gross  motor  movements 
measured  in  terms  of  minutes  of 
conventional  instruction. 

A five  point  rating  of  the  simplicity 
or  complexity  of  the  performance  element. 
Simple  psychomotor  content  (one)  includes 
gross  motor  movements . Complex  psycho- 
motor content  (five)  involves  fine  motor 
dexterity,  complex  eye-hand  coordination, 
and  precise  motor  movements. 
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TABLE  2 

BEHAVIOR  CLASSIFICATION  DEFINITIONS 


BEHAVIOR  CATEGORY 

Discriminated  Recall -Easy  Difficult 


Cl  assi fi cati on-Easy/Di ff i cult 


Rule  Using-Easy/Difficult 


Higher  Order  Rule  Using 
Easy/Difficult 


Psychomotor-Easy/Di ff i cult 


DEFINITION 

Given  a symbol,  object  or  event,  a 
student  responds  by  providing  the 
associated  member  of  the  response  set. 

If  the  number  of  members  in  the  stimulus 
set  was  large  and/or  if  the  members  of 
the  response  set  were  unfamiliar  (e.g., 
new  technical  terms),  the  objective  was 
judged  to  be  difficult. 

Given  a new  example  from  a class  of  items, 
a student  responds  with  class  membership. 

The  easy/difficult  dimension  was  based 
on  the  number  of  classes. 

Given  a new  experience,  a student  applies 
a rule  (a  set  of  operations)  to  produce 
the  appropriate  members  of  the  response 
set.  Rule  Using-Easy  was  essentially 
following  written  procedures  (e.g.,  technical 
orders).  Rule  Usinq-Difficul t was 
following  procedures  in  the  absence  of 
written  guidance. 

Analogous  to  problem,  sol ving  activity,  the 
retrieval  or  generation  of  rules  by  applying 
a higher  order  rule  to  determine  problem 
class  membership.  The  easy/difficult 
dimension  was  based  on  the  number  of  steps 
in  the  problem  solution. 

The  performance  of  fine  or  gross  motor 
control  movements  to  accomplish  a task. 

The  easy/difficult  dimension  was  based  on 
the  familiarity  or  novelty  of  the  task 
components. 
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The  psychologist  raters,  although  familiar  with  the  AIS  courses,  were  by  no 
means  subject  matter  experts.  Therefore,  their  ratings  were  checked  and,  in  a few 
cases,  modified,  by  members  of  the  course  materials  development  teams.  This 
procedure  was  undertaken  as  an  attempt  to  provide  at  least  face  validity  for  these 
ratings.  Since  independent  ratings  of  the  objectives  were  not  obtained,  interrater 
reliability  estimates  could  not  be  generated. 

4. 2. 2. 2 Instructional  Design  Variables  - As  was  previously  discussed,  AIS,  at  the 
time  of  this  study,  included  insufficient  examples  of  lessons  taught  by  media 
other  than  programmed  texts  to  include  alternative  media  as  variables  in  the 
predictive  model.  A similar  situation  existed  with  respect  to  alternative 
instructional  treatments  designed  to  compensate  for  specific  deficiencies.  Instruc- 
tional design  considerations  were,  therefore,  limited  to  "organizers."  Organizers 
were  defined  as  "assists"  provided  in  the  instructional  material  to  enhance  the 
trainee's  achievement  of  a behavioral  objective. 

The  most  prevalent  type  of  organizers  in  the  AIS  materials  was  the  list  of 
objectives  at  the  beginning  of  each  lesson.  Since  every  lesson  included  such  a 
list,  there  was  no  variability  across  lessons  with  respect  to  this  factor;  and 
it  was,  therefore,  of  no  value  as  a predictor  variable. 

Two  other  types  of  organizers  were  present  which  did  vary  from  one  lesson  to 
another  - embedded  questions  and  self-check  items.  With  the  exception  of  those 
lessons  which  were  strictly  performance  oriented,  embedded  questions  were  scattered 
throughout  the  textual  material.  Each  consisted  of  a multiple-choice  item,  the 
correct  answer  to  which  was  provided  on  the  last  page  of  the  lesson.  The  majority 
of  the  AIS  lessons,  again  excluding  performance  lessons,  included  a self-check  item 
to  be  completed  following  study  of  the  lesson  but  prior  to  the  lesson  test.  These 
items  were  also  multiple-choice.  The  student  marked  his  response  on  chemically 
treated  paper  which  then  revealed  whether  or  not  the  response  was  correct. 

The  number  of  embedded  questions  and  the  number  of  self-check  items  were 
recorded  for  each  of  the  AIS  lesson  analyzed.  If  no  items  of  particular  type  were 
present,  a zero  was  recorded.  Although  this  information  was  taken  directly  from 
existing  AIS  lessons,  a tactic  which  would  not  be  possible  if  the  model  were 
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actually  being  used  to  predict  the  characteristics  of  a self-paced  course  yet  to  be 
constructed,  the  number  of  embedded  and  self-check  items  was  assumed  to  reflect 
instructional  design  policies  which  could  and  should  be  established  prior  to  actual 
materials  development. 

4. 2. 2. 3 Test  Characteristics  - Consideration  of  test  characteristics  was  limited 
to  the  lesson  tests  themselves.  Due  to  the  inclusion  of  new  material  in  the  AIS 
lessons,  there  were  no  examples  of  complete  blocks.  Thus,  all  of  the  block  tests 
included  some  items  from  lessons  which  were  not  sampled  for  this  study. 

Three  test  characteristics  were  measured  for  each  lesson  test.  For  each  lesson 
which  included  a multiple-choice  test,  the  number  of  test  items  and  the  test 
criterion  (expressed  as  a percentage)  were  recorded.  For  lessons  without  multiple- 
choice  tests,  a zero  was  recorded  for  both  number  of  items  and  criterion.  For  each 
lesson  which  included  a performance  test,  a situation  in  which  the  student  was  to 
perform  a task  to  the  instructor's  satisfaction,  the  number  of  specific  checkpoints 
in  the  performance  task  was  recorded.  Again,  a zero  was  recorded  for  this  variable 
if  the  lesson  did  not  include  a performance  test. 

4. 2. 2. 4 Student  Characteristi cs  - The  variables  chosen  as  student  characteristics 
were  determined  on  the  basis  of  (a)  the  ease  with  which  they  could  be  .obtained  in  a 
variety  of  military  training  environments,  (b)  how  representative  they  were  of  a 
wide  variety  of  individual  differences,  both  cognitive  and  affective,  and  (c)  their 
known  relationship  to  the  prediction  of  AIS  training  time  criteria  in  different 
types  of  training  courses.  Thus,  the  resulting  student  characteri sti c variables 
include  the  four  primary  composite  scores  from  the  Armed  Services  Vocational 
Aptitude  Battery  (ASVAB),  sex,  and  selected  affective  and  cognitive  variables 
which  were  common  to  the  AIS  pre-assessment  batteries  of  both  the  IM  and  WM 
courses. 

A complete  description  of  each  of  the  selected  student  characteri sti c variables 
is  presented  in  Table  3.  The  variables  are  organized  in  three  categories:  (a)  Armed 

Services  Vocational  Aptitude  Battery  (ASVAB)  Variables  and  sex;  (b)  Cognitive  AIS 
Pre-Assessment  Battery  Variables;  and  (c)  Affective  AIS  Pre-Assessment  Battery 
Variables.  Wherever  applicable,  the  references  for  a particular  measure  of  student 
characteristic  variables  are  cited. 
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TABLE  3 

STUDENT  CHARACTERISTIC  VARIABLES 

VARIABLE  CATEGORY  VARIABLE  NAMES  DEFINITION 
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I.  ASVAB  Composites 
and  Sex 


II . Cognitive  AIS 
Pre-Assessment 
Variables 


III.  Affective  AIS 
Pre-Assessment 
Variables 


ASVAB  Scores: 

General  Ability 

Electrical  Ability 

Administrative  Ability 

Mechanical  Ability 


A standard  vocational  aptitude 
battery,  administered  to  all 
enlisted  inductees,  which 
contains  four  primary  composite 
scores  for  general,  electrical, 
administrative,  and  mechanical 
abilities,  respectively. 


Sex  An  identifier  of  individual 

differences  in  training  times 
attributable  to  sex  differences, 
where  males  = 1,  and  females  = 0. 


Reading  Vocabulary  A 45-item  measure  of  students' 

Test  - Total  Scores  ability  to  recognize  the  defini- 

tion of  terms  frequently  used  in 
Air  Force  documents  (Diegnan,  1973) 


Reading  Skills  Scales  Course-specific  measures  of 
1 and  2 students'  reading  comprehension 

and  speed  on  materials  extracted 
from  IM  and  WM  technical  manuals. 
Scales  1 and  2,  consisting  of  10 
items  each,  differ  only  in  terms 
of  difficulty,  with  longer  pas- 
sages and  more  rigid  time  limits 
imposed  on  Scale  2 (McCombs,  1974). 


Concealed  Figures  Test  A 20-item  measure  of  students 

ability  to  make  perceptual  dis- 
tinctions by  recognizing  which 
of  five  simple  geometrical  figures 
is  embedded  within  a complex 
pattern  (Diegnan,  1974). 


Pre-Course  State  A 20-item  self-report  measure 

Curiosity  Scale  of  students'  anticipated  feelings 

of  interest  in  learning  the  IM 
or  WM  course  materials  (Leherissey- 
McCombs,  1971). 


Pre-Course  State  A 20-item  self-report  measure  of 

Anxiety  Scale  students'  anticipated  feelings  of 

tension  or  apprehension  about 
learning  the  IM  or  WM  course 
materials  (Spielberger,  Gorsuch, 

& Lus‘'»ne , 1 970) . 
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TABLE  3 (cont.  ) 

STUDENT  CHARACTERISTIC  VARIABLES 


VARIABLE  CATEGORY 

III.  Affective  AIS 
Pre-Assessment 
Variables  (cont.) 


VARIABLE  NAMES 

Test  Anxiety 
Questionnai re 


Internal -External 
Scale 


General  Media 
Preference  and 
Instructional 
Experience  Subscales: 
Audio  Preference 

Visual  Preference 

Printed  Preference 

Experience  with  Con- 
ventional Institution 

Experience  with  Self- 
Paced  Instruction 


DEFINITION 


A 16-item  self-report  measure  of 
students'  tendencies  to  become 
anxious  when  taking  ability  or 
achievements  tests  (Sarason,  1958). 

A 29-item  self-report  measure 
of  students'  tendencies  to 
feel  in  control  of  events 
versus  controlled  by  external 
events  (Rotter,  1966). 

A 15-item  self-report  scale, 
consisting  of  five  subscales 
which  measure  students'  general 
preference  for  audio  versus 
printed  materials,  as  well  as 
their  reported  degree  of 
experience  with  conventional 
versus  self-paced  instruction 
(McCombs,  1974) 
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The  primary  rationale  for  the  selection  of  the  four  ASVAB  variables  and  sex 
was  that  these  variables  are  easily  obtained  on  enlisted  trainees  in  all  branches 
of  the  military  services.  In  addition,  many  years  of  systematic  research  with  the 
ASVAB  battery  has  shown  it  to  be  a highly  reliable  and  valid  measure  of  individual 
differences  in  vocational  aptitudes.  A number  of  research  studies,  as  well  as  our 
own  experience  with  AIS  data,  have  shown  sex  to  be  related  to  individual  differences 
in  training  times. 

Within  the  Cognitive  AIS  Pre-Assessment  Battery  Variables  category,  three 
reading  ability  measures  and  one  perceptual  reasoning  ability  measure  were  chosen. 
The  reading  variables  have  been  shown  to  be  highly  predictive  of  independent 
variance  in  training  time  criteria  in  each  of  the  four  AIS  courses.  In  addition, 
it  was  felt  that,  even  though  these  particular  reading  measures  might  not  be 
available  in  other  military  applications,  similar  reading  vocabulary,  comprehensive, 
or  speed  measures  we>^e  likely  to  be  available.  The  perceptual  reasoning  variable, 
defined  by  scores  on  the  Concealed  Figures  Test,  was  selected  on  the  basis  that  it 
was  available  for  both  IM  and  WM  students,  and  it  represented  a perceptual /reasoni ng 
ability  that  would  be  generally  related  to  student  performance  in  a variety  of 
technical  specialties.  At  the  time  this  variable  was  selected,  preliminary  AIS  data 
indicated  that  it  was  moderately  related  to  training  times  on  some  of  the  IM  course 
materials. 

The  particular  measures  chosen  with  the  Affective  AIS  Pre-Assessment  Variable 
category  were  chosen  on  the  basis  of  (a)  their  known  relationship  to  AIS  training 
time  criteria;  (b)  the  wide  range  of  student  motivations,  attitudes,  and  learning 
style  preferences  they  represented;  and  (c)  their  availability  as  common  predictor 
variables  in  both  the  IM  and  WM  courses.  For  example,  state  curiosity,  state  anx- 
iety, and  test  anxiety  have  all  been  shown  to  be  highly  predictive  of  AIS  training 
time  criteria  (McCombs  and  Siering,  1976).  Although  at  the  time  of  this  study  not 
much  AIS  data  were  available  of  the  Internal -External  Scale,  it  was  judged  to  be  a 
student  characteristic  variable  that  would  be  expected  to  be  highly  related  to 
student  training  times  in  self-paced  instructional  environments.  Similarly, 
students'  preferences  for  audio  versus  visual  printed  instructional  modes  of  learn- 
ing, and  their  experience  with  self-paced  versus  conventional  instruction  were  also 
judged  to  be  important  affective  predictor  variables  in  self-paced  instructional 
environments. 
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4.2.3  Creation  of  Files  for  Data  Analysis  - Creating  the  data  files  was  a lengthy 
process  which  involved  merging  three  distinct  classes  of  data: 

0 Student  data  available  from  the  AIS  data  analysis  system:  Course;  Social 

Security  Number  (used  by  AIS  as  an  ID);  sex;  preassessment  test  data; 
block  and  lesson  number;  module  number  (i.e.,  instructional  alternative 
type);  first-attempt-lesson  time;  and  first-pass-block  time 

0 Supplementary  student  data:  The  ASVAB  General,  Electrical,  Administrative, 

and  Mechanical  scores 

0 Lesson  characteristic  data:  the  POI  derived  quantitative  measures;  content 

classification  measures;  behavior  classification  measures;  instructional 
design  measures;  and  test  characteristic  measures. 

Student  data  were  extracted  from  AIS  files  representing  the  nine  blocks  of 
the  two  courses  using  the  AIS  Data  Extraction  Program  (DEP).  It  was  necessary  to 
extract  data  from  separate  and  distinct  time  periods  in  order  to  avoid  time 
periods  in  which  the  courses  were  undergoing  substantive  changes.  The  product  of 
this  activity  was  nineteen  separate  files. 

ASVAB  scores  were  not  normally  recorded  in  the  AIS  files  since  they  were 
not  usually  available  early  enough  in  the  course  to  be  of  value  as  predictors. 

For  purposes  of  this  study,  however,  ASVAB  scores  were  obtained  from  Air  Training 
Command's  Training  Management  Information  System  (TRAMIS)  for  as  many  of  the  students 
as  possible  represented  in  the  DEP  files  described  above.  The  scores  were  obtained 
on  cards,  input  into  the  computer,  and  an  additional  file  was  constructed. 

A total  of  64  lessons  was  sampled.  Twenty-nine  of  these  were  from  the  IM 
course;  six  from  block  one;  six  from  block  two,  six  from  block  three,  four  from 
block  four;  an^  seven  from  block  five.  The  remaining  35  were  drawn  from  the  WM 
course:  nine  from  block  nine;  eight  from  block  ten;  ten  from  block  eleven;  and 

eight  from  block  twelve.  Lesson  characteristic  data  were  punched  on  cards,  input, 
and  a file  was  created  for  each  of  the  nine  blocks. 
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A special  purpose  program  was  written  to  merge  files  across  categories.  Each 
run  of  this  program  produced  a file  containing  all  relevant  data  for  each  of  the 
nineteen  DEP  files  previously  created.  These  nineteen  files  were  then  merged  into 
a single  tape  file  containing  approximately  13,700  records  where  each  record 
represented  a particular  student's  first  attempt  on  a lesson  of  known  characteristics. 
That  is,  each  record  contained  the  relevant  characteristics  of  both  the  student  and 
the  lesson  as  well  as  the  time  required  for  his  first  attempt  on  the  lesson. 

The  magnitude  of  this  file  made  its  retention  and  manipulation  on  a disk 
impractical.  The  study  required  two  comparable  samples,  one  for  regression 
equation  generation  and  a second  for  equation  validation.  Therefore,  the  file  was 
immediately  split  into  two  samples  on  the  basis  of  whether  the  final  digit  of  a 
trainee's  Social  Security  Number  was  odd  or  even.  After  excess  records  had  been 
deleted  from  the  larger  sample,  each  sample  contained  6849  records. 

Missing  and  incorrect  data  were  rampant  throughout  the  developing  AIS  data 
base.  Typical  problems  included  preassessment  forms  which  were  incorrectly  read 
by  the  system's  optical  scanner,  and  trainees  who  were  enrolled  without  completing 
, any  of  the  preassessment  tests.  ASVAB  scores  obtained  from  TRAMIS  were  often 
'incomplete  or  missing  entirely.  First-attempt-lesson-times  contained  a small  but 
persistent  number  of  widely  deviant  values.  The  primary  cause  of  these  outliers 
was  miscoding,  e.g.,  if  a student  worked  on  a lesson  during  two  consecutive  days, 
the  number  of  days  was  often  mismarked  on  the  test  answer  sheet.  The  measure  of 
first-pass-block-time  was  also  error  prone.  It  was  defined  as  the  elapsed  time 
from  starting  the  first  lesson  in  the  block  until  the  block  test  was  completed. 

Errors  such  as  mismarked  dates  often  made  this  value  fallacious. 

To  reduce  these  errors  as  much  as  possible,  sets  of  constraints  were  established 
for  these  variables.  Preassessment  scores  were  recoded  as  missing  values  if  they 
were  actually  missing  or  exceeded  the  known  limits  of  the  test  scores.  All  ASVAB 
scores,  if  present,  were  determined  to  be  reasonable.  First-attempt-lesson-times  were 
limited  to  a range  of  fifteen  to  three-hundred  minutes.  This  eliminated  approxi- 
mately five  percent  of  the  data  from  either  end  of  the  distribution.  Block  times 
were  excluded  if  the  reported  time  was  not  greater  than  the  sum  of  the  first-attempt- 
lesson-times  or  if  the  reported  time  fell  outside  the  range  established  as 
reasonable  by  prior  experience  with  AIS. 
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The  regression  analyses  was  conducted  using  the  (SPSS)  Statistical  Package 
for  the  Social  Sciences  (Nie,  Hull,  Jenkins,  Steinbrenner,  and  Brent,  1975).  SPSS 
offers  two  options  with  regard  to  missing  data:  "pair-wise  deletion"  which 

includes  all  of  the  data  points  available  for  a particular  variable  in  the  analysis, 
and  "list  or  case-wise  deletion"  which  excludes  any  case  from  consideration  if  that 
case  has  one  or  more  missing  data  poitits  in  any  of  the  variables  included  in  the 
analysis.  Although  the  former  option  is  less  expensive  in  terms  of  lost  data, 
it  tends  to  result  in  fallaciously  high  correlation  coefficients.  Therefore,  the 
case-wise  deletion  option  was  selected.  This  reduced  Sample  One,  the  data  base 
to  be  used  for  regression  equation  generation,  to  2974  cases,  each  having  a complete 
set  of  data  points  for  each  variable.  It  should  be  noted  that  this  figure  does  not 
represent  2974  different  students.  Varying  numbers  of  observations  were  generated 
by  individual  students.  For  example,  data  generated  by  a particular  lii  student  might 
be  present  for  the  data  base  for  three  lessons  in  Block  One  and  two  lessons  in 
Block  Two.  The  implications  of  these  repeated  measures  are  discussed  in  paragraph 
4.3.1  below. 

A summarizing  list  of  the  variables  investigated,  their  means  and  standard 
deviations  (based  on  the  final  2974  cases),  and  their  correlation  with  the  criterion 
variable  is  shown  in  Table  4.  Given  the  size  of  the  sample,  first  order  correlations 

equal  to  or  greater  than  R = +.05  are  significant  (p  < .05). 

4.3  STATISTICAL  ANALYSIS  AND  RESULTS 

4.3.1  Description  of  Statistical  Methods  - The  data  were  analyzed  using  several 
statistical  methods,  primarily  programs  from  the  Statistical  Package  for  the 

Social  Sciences  (Nie  et  al  1975).  Available  SPSS  regression  program  options  were 

.sed  to  constrain  data  values  to  reasonable  limits  and  to  screen  out  missing  data. 

The  two  primary  methods  employed  in  obtaining  the  final  prediction  equation  were 
linear  model  comparisons  (Ward  and  Jennings,  1973)  and  stepwise  multiple  regres- 
stion  analysis. 

In  linear  model  comparisons,  two  linear  models  (i.e.,  regression  equations) 

are  compared  in  terms  of  the  reduction  of  the  multiple  correlation  coef-^icient 
2 

squared  (R  ).  The  model  with  the  larger  number  of  predictors  is  referred  to  as 
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MEANS,  STANDARD  DEVIATIONS,  AND  FIRST  ORDER  CORRELATIONS 
PREDICTOR  AND  CRITERION  VARIABLES 

OF 

VARIABLES 

MEAN 

STD. 

DEV. 

R WITH 
CRITERION 

COURSE  CONTENT  VARIABLES 
Quantitative  Measures  (from  POI) 

Lecture/ Di scussi on  Minutes 

89.54 

97.58 

.489 

Demonstration/Performance  Minutes  - 

Cogni ti ve 

55.70 

64.45 

.488 

Demonstration/Perfonnance  Minutes  - 

Psychomotor 

27.91 

63.19 

-.294 

Programmed  Instruction/AV  Minutes 

8.43 

38.67 

. 047 

Outside  Assignment  Minutes 

51.27 

127.10 

. 029 

Content  Classification  Measures 

Memory  Content  Minutes 

68.86 

71  .99 

.321 

Simple/Complex  Memory  Rating  (1-5) 

2.6i 

0.74 

.201 

Cognitive  Content  Minutes 

145.67 

147.37 

.329 

Simple/Complex  Cognitive  Rating  (1-5) 

2.81 

1.05 

-.015 

Psychomotor  Content  Minutes 

20.95 

53.13 

-.084 

Simple/Complex  Psychomotor  Rating  (1-5) 

1.23 

1.90 

-.239 

Behavior  Classification  Measures  (All 
Val ues  in  Minutes  ) 

Discriminated  Recall  - Easy 

86.13 

126.06 

.102 

Discriminated  Recall  - 

Difficult 

25.88 

58.08 

.049 

Classification  - Easy 

1.02 

8.72 

-.079 

Classification  - Difficult 

(no  instances 

of  this 

category) 

Rule  Using  - Easy 

86.56 

111.85 

.368 

Rule  Using  - Difficult 

4.18 

19.43 

.138 

Higher  Order  R.U.  - Easy 

13.13 

37.00 

.349 

Higher  Order  R.U.  - Difficult 

1.57 

11.82 

-.099 

Psychomotor  - Easy 

14.30 

36.23 

-.122 

Psychomotor  - Difficult 

(no  instances 

of  this 

category ) 

INSTRUCTIONAL  DESIGN  VARIABLES 

No.  of  Embedded  Questions 

16.27 

1 5.18 

.590 

No.  of  Self-Check  Items 

2.74 

4.54 

-.149 

TEST  CHARACTERISTIC  VARIABLES 

No.  of  Lesson  Mastery  Test  Items 

9.38 

6.69 

.478 

Master  Test  Criterion  (Percent) 

57.70 

29.75 

.288 

No.  of  Performance  Test  Check  Points 

8.26 

19.04 

-.279 
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TABLE  4 


MEANS,  STANDARD  DEVIATIONS  AND  FIRST  ORDER  CORRELATIONS  OF 
' PREDICTOR  AND  CRITERION  VARIABLES 


(Cont ' d) 


VARIABLES 

MEAN 

STD. 

DEV. 

R WITH 
CRITERION 

STUDENT  CHARACTERISTIC  VARIABLES 

Sex  (0  = Female,  1 = Male) 

0.87 

0.33 

-.233 

ASVAB  Scores 

General 

68.85 

13.30 

-.089 

Admi ni strati ve 

59.62 

18.47 

.149 

Mechani cal 

58.97 

25.88 

- .384 

El ectri cal 

65.98 

1 6.65 

-.252 

Cognitive  AIS  Preassessment  Scores 

Reading  Vocabulary 

22.37 

6.80 

-.041 

Reading  Skills  Scale  1 

5.38 

1.80 

-.160 

Reading  Skills  Scale  2 

5.18 

2.18 

.218 

Concealed  Figures 

7.96 

3.60 

- .042 

Affective  AIS  Preassessment  Scores 

State  Curiosity 

65.76 

7.77 

-.049 

State  Anxiety 

38.89 

8.36 

.084 

Test  Anxiety 

30.11 

, 7.57 

.182 

Internal -External 

14.43 

3.88 

- .021 

Audio  Preference 

11.09 

3.C4 

-.007 

Visual  Preference 

7.89 

2.22 

-.147 

Printed  Preference 

4.39 

1.44 

.069 

Conventional  Instruction  Experience 

6.94 

1.64 

- .012 

Self-Paced  Instruction  Experience 

5.45 

1.35 

- .095 

CRITERION  VARIABLE 

First-Attempt-Lesson  Time 

84.90 

64.74 

- 
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the  full  model,  and  the  model  with  the  smaller  number  of  predictors  is  called  the 

2 

restricted  model.  The  reduction  in  R is  F distributed  with  degrees  of  freedom 
df^j/df^  according  to  the  following  equation. 

^ _ (r/  - / (df)^ 

(1  - r/)  / (df)2 

2 2 

where  R,  = R of  the  full  model, 

^2  2 

R^  = R of  the  restricted  model, 

(df)-]  = n^  - n^,  the  difference  between  the  number  of  linearly 
independent  predictor  vectors  of  the  full  (n^)  and 
restricted  (n^)  models,  and 
(df)^  = N - n^  where  N is  the  total  number  of  cases. 

For  the  linear  model  comparisons  in  this  study,  SPSS  regression  program  options 

were  used  to  delete  predictor  variables  and  to  generate  composite  predictors.  The 
2 

R values  were  obtained  from  the  SPSS  program,  anu  the  F values  were  obtained  from 
a FORTRAN  program  produced  by  the  AIS  project  staff.  Stepwise  multiple  regression 
analysis  is  simply  multiple  reg>^ession  analysis  performed  by  adding,  step  by  step, 
the  predictor  variable  which  yields  th  ’ highest  F value  for  the  increase  in  R^ 
according  to  the  above  equation.  The  SPSS  multiple  regression  programs  include  a 
stepwise  option  which  was  employed  where  appropriate  for  this  study. 

The  models  employed  in  this  series  of  linear  model  comparisons  did  not  control 
for  the  effect  of  repeated  measures  on  the  lessoh  variables.  There  are  several 
reasons  why  this  was  not  done.  First,  one  controls  for  the  effect  of  a particular 
variable,  e.g.,  subjects,  to  increase  the  precision  of  an  F-test  between  two  models. 
The  very  large  n_  in  this  study  provided  sufficient  precision  so  that  additional 
control  over  extraneous  error  variance  was  not  necessary.  In  general,  the  F-tests 
resulting  from  model  comparisons  were  highly  significant.  Second,  the  effects 
of  the  repeated  measures  were  curtailed  by  the  fact  that  the  repeated  observations 
for  a given  subject  were  for  different  lessons  (different  learning  tasks)  rather 
than  on  the  same  lesson  and  that  subjects  with  repeated  measures  varied  with 
regard  to  the  sets  of  lessons  on  which  the  measures  were  repeated.  Third,  to  con- 
trol for  the  repeated  measures  would  have  required  the  generation  of  a binary 
vector  to  represent  each  individual  student.  This  would  have  far  exceeded  the 
capacity  of  the  SPSS  program.  Finally,  it  must  be  remembered  that  the  purpose 
• • 4-24 
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of  the  resultant  regression  model  was  to  predict  first-attempt-lesson-time  given 
only  the  characteristics  of  the  lesson  and  of  a generally  defined  student  population. 
Thus,  the  user  of  the  prediction  equation  would  have  no  means  by  which  to  identify 
individual  students. 

Cross  validation  of  the  final  regression  equation  model  also  employed  the 
SPSS  regression  program.  SPSS  regression  program  options  were  used  to  define  a 
variable,  predicted  first-attempt-lesson-time,  which  was  the  result  of  the  applica- 
tion of  the  final  regression  equation  to  data  from  Sample  Two.  The  values  of  this 
variable  were  than  correlated  with  actual  Sample  Two,  first-attempt-lesson-times. 

4.3.2  Elimination  of  Two  Variables  Highly  Correlated  with  the  Criterion  - Early 
analyses  indicated  the  presence  of  two  variables  which  were  highly  correlated  with 
the  criterion  of  f i rst-attempt-lesson-time.  These  were  the  instructional  design 
variables.  Number  of  Embedded  Questions  (r  = .590),  and  the  test  characteristic 
variable.  Number  of  Lesson  Test  Items  (r  = .478).  Since  these  variables  were  also 
highly  correlated  with  a number  of  other  variables,  they  tended  to  mask  the 
relevance  of  these  potential  predictors  and  accounted  for  more  of  the  variability 
than  was  logically  justified. 

It  will  be  recalled  that  the  numbers  of  Embedded  Questions  and  Lesson  Test  Items 
were  taken  directly  from  existing  AIS  materials.  This  action  was  justified  on  the 
assumption  that  the  numbers  of  items  reflected  instructional  design  decisions 
which  could  be  made  prior  to  the  development  of  self-paced  materials.  Unfortunately 
for  this  study,  AIS  material  developers  employed  a consistent  policy  with  regard 
to  the  number  of  embedded  and  lesson  test  questions  throughout  their  instructional 
development  effort  in  both  of  the  courses  sampled.  Thus,  almost  all  of  the 
variability  in  these  two  variables  could  be  traced  directly  to  the  amount  of  subject 
matter  content  in  each  lesson  and,  hence,  its  average  first-attempt-time.  Since  the 
current  study  was  basically  a survey,  and  direct  manipulation  of  the  various  pre- 
dictor variables  was  not  feasible,  some  problems  of  this  type  had  to  be  expected. 
Despite  the  resultant  loss  of  predictability,  it  was  imperative  that  these  two 
variables  be  deleted  from  the  predictor  set  in  order  to  obtain  a realistic  assess- 
ment of  the  relevance  of  the  remaining  variables. 
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Two  linear  model  comparisons  were  made  in  eliminating  these  two  variables.  The 
full  regression  model,  prior  to  elimination  of  the  variables,  contained  34 
predictors  - all  of  those,  other  than  the  behavior  classification  measures,  are 
shown  in  Table  4.  The  multiple  correlation  coefficient  obtained  was  .68599,  R = 
.47058.  Elimination  of  the  Embedded  Questions  variable  reduced  proportion  of  the 

p 

variance  accounted  for  to  R = .42525.  A test  for  the  significance  of  the  reduction 
in  R^  resulted  in  an  F (1,  2939)  = 251.643,  ^ < .001.  When  the  variable  Number  of 
Lesson  Test  Items  was  removed,  R^  was  further  reduced  to  .40413,  F_  (1,  2940)  = 
108.034,  £ < .001. 

4.3.3  Reduction  of  Predictor  Set  Through  Linear  Models  Comparisions  - A series  of 
linear  model  comparisons  was  made  in  order  systematically  to  obtain  a more  parsi- 
monious set  of  predictor  variables.  Although  the  specific  questions  of  Interest 
could  have  been  asked  by  isolating  the  relevant  variables,  it  was  considered  more 
appropriate  that  the  comparisons  be  made  in  the  context  of  the  complete  set  of 
remaining  predictors.  In  all  of  the  comparisons  to  be  discussed,  emphasis  was 
placed  on  cost-effectiveness  rather  than  on  statistical  significance.  Due  to  the 
magnitude  of  the  data  base,  almost  all  of  the  linear  model  comparisons  resulted  in 
significant  differences.  In  general,  however,  the  increased  cost  of  obtaining 
data  required  by  the  additional  variables  in  the  full  model  did  not  appear  to  be 
justified  by  the  increment  in  predictability  provided  by  the  full  model.  Therefore, 

the  decision  rule  adopted  was  to  select  the  restricted  model  unless  the  reduction 
2 

in  R approached  or  exceeded  .01000. 

4. 3.3.1  Course  Content  and  Behavior  Classification  Measures  - An  obvious  first 
step  in  reducing  the  variable  set  was  to  select  one  of  the  two  approaches  for 
classifying  course  content.  Prior  to  comparing  the  two  approaches,  however,  the 
extent  to  which  the  number  of  variables  in  each  set  could  be  reduced  was  investi- 
gated. The  content  classification  scheme  was  attacked  first. 

The  initial  full  model.  Model  1,  contained  the  32  variables  remaining  after 

Number  of  Embedded  Questions  and  Number  of  Lesson  Test  Items  had  been  deleted  and 

excluding  the  behavior  classification  variables.  The  multiple  correlation 

2 

generated  by  Model  1 was  .63572,  R = .40413. 


4-26 


/MCOO/W/Vrf.1.  DOUGLAS  ASTRONAUTICS  COAfRANT  • mAST 


V 


EVALUATING 
INSTRUCTIONAL  SYSTEMS 

The  first  question  asked  concerned  whether  the  simplicity/complexity  dimension 
employed  by  this  approach  added  substantially  to  the  scheme's  predictive  utility. 

A restricted  model,  Model  2,  was  defined  from  Model  1 by  deleting  the  following 
three  rating  scale  variables:  Memory  - Simple/Complex;  Cognitive  - Simple/Complex ; 

and  Psychomotor  - Simple/Complex.  Model  2 resulted  in  an  R of  .38395,  a reduction 
from  Model  1 of  .02018,  F (3,  2941)  = 33.200,  £ < .001.  This  was  considered  to  be 
too  great  a loss  in  predictability  for  the  cost  of  obtaining  the  simple/complex 
ratings,  and  the  decision  was  made  to  retain  the  rating  scales,  i.e..  Model  1. 

Attention  was  then  directed  to  the  behavior  classification  approach.  A full 
model.  Model  3,  was  defined  which  consisted  of  34  predictor  variables.  Model  3 
differed  from  Model  1 in  that  the  eight  behavior  classification  measures 
were  substituted  for  the  six  content  classification  measures.  As  was  noted  in 
Table  4,  the  lessons  sampled  provided  data  for  only  eight  of  the  ten  behavior 
classification  measures.  No  instances  of  Difficult  Classification  or  Difficult 
Psychomotor  tasks  were  encountered. 

Model  3 resulted  in  an  of  .35057  as  compared  with  an  R^  of  .40413  for 
Model  1.  Thus,  it  was  immediately  obvious  that  the  content  classification  approach 
was  preferable  in  terms  of  level  of  predictability.  The  proposed  questions  of 
interest  regarding  the  behavior  classification  approach  were  conducted  anyway. 

The  first  question  asked  was  similar  to  that  asked  regarding  the  content 

classification  scheme  _ could  the  easy/difficult  dimension  be  dropped  without 

substantive  loss  in  predictability?  A restricted  model.  Model  4,  was  defined  in 

which  the  eight  available  behavior  classi fication  predictor  variables  were  reduced 

to  five.  The  values  of  Discriminated  Recall -Easy  and  -Difficult,  were  added 

together  to  produce  a simple  Discriminated  Recall  variable.  The  same  procedure 

was  repeated  with  respect  to  Rule  Using-Easy  and  -Difficult  and  Higher  Order  Rule 

2 

Using-Easy  and  -Difficult.  Model  4 resulted  in  an  R of  .35451,  a reduction  from 
Model  3 of  .00606,  F_  (3,  2939)  = 9.285,  p < .001.  Since  the  reduction  in  R^  was 
substantially  less  than  0.01,  deletion  of  this  dimension  would  appear  justified  in 
view  of  the  difficulty  of  obtaining  the  easy/di fficul t ratings  and  their  probable 
unrel  iabil ity. 
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A second  question  was  concerned  with  the  utility  of  the  behavior  category 
Higher  Order  Rule  Using.  Re^itively  few  of  the  lesson  samples  contained  examples 
of  this  type  of  behavior.  It  was  thought  that  this  might  well  be  a typical 
circumstance  in  military  technical  training  where  even  most  troubleshooting  tasks 
have  largely  been  reduced  to  a procedure -foil owing  activity.  Therefore,  a further 
restricted  model,  Model  5,  was  defined  in  which  the  values  of  the  variables  Higher 
Order  Rule  Using  and  Rule  Using  were  added  together  to  produce  a single  Rule  Usi ng 
variable.  Model  5 resulted  in  an  R of  .35443,  a reduction  from  Model  4 of  only 
.00008,  £ (1,  2942)  = 0.365,  p = .546.  This  was  obviously  not  a sufficiently 
substantive  loss  in  predictability  to  justify  generation  of  the  two  separate 
categories  Rule  Using  and  Higher  Order  Rule  Using. 

As  was  mentioned  above,  the  full  behavior  classification  model  (Model  3) 
resulted  in  a smaller  R than  did  the  final  content  classification  model  (Model  1). 
This  was  despite  the  fact  that  Model  1 contained  fewer  predictor  variables  (32) 
than  did  Model  3 (34).  An  F test  was  therefore  inappropriate  as  well  as  unnecessary 
and  Model  1 was  selected.  That  is,  the  six  content  classification  variables  were 
retained  as  predictors,  and  the  behavior  classification  variables  were  deleted 
from  the  predictor  set. 

Of  the  two  approaches,  the  content  classification  scheme  was  also  considered 
more  desirable  on  practical  grounds.  It  will  be  recalled  that  the  content  ratings 
were  generated  by  course  materials  development  and  instructional  personnel,  the 
type  of  personnel  who  would  be  most  readily  available  to  rate  the  content  of  a 
course  being  considered  for  self-pacing.  The  more  esoteric  behavior  classification 
scheme  was  conducted  by  instructional  psychologists  familiar  with  the  behavioral 
concepts  and  terminology.  It  can  be  assumed  that  more  extensive  rater  training 
would  be  required  if  course  personnel  were  required  to  produce  the  behavior 
ratings. 

4. 3. 3. 2 Quantitative  Course  Content  Measures  - It  will  be  recalled  that  the  time 
allocated  to  the  conventional  lesson  by  the  POI  was  broken  down  into  five  categories 
Lecture/Discussion;  Demonstration/Performance-Cogni tive ; Demonstration/Performance- 
Psychomotor;  Programmed  Instruction/AV;  and  Outside  Assignment.  The  question  of 
interest  concerned  the  necessity  of  these  five  categories.  The  most  likely 
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candidate  for  reduction  appeared  to  be  the  Cognitive  and  Psychomotor  subsets  of 
Demonstrati  on/ Performance  time.  The  model  resulting  from  the  course  content 
comparisons,  Model  1,  was  employed  as  the  full  model.  The  restricted  model. 

Model  6,  differed  from  Model  1 in  that  it  included  only  a single  Demonstration/ 
Performance  variable,  and  was  generated  by  summing  the  values  of  the  Demonstration/ 
Performance-Cognitive  and  -Psychomotor  variables  in  Model  1.  Model  6 resulted  in 
R of. 38800,  a reduction  from  Model  1 of  .01613,  £ (1,  294)  = 79.612,  p < .001. 

The  cost  of  generating  these  two  sub-categories  was  considered  justified  by  the 
increased  predictability.  Model  1 was  again  retained. 


The  overall  utility  of  breaking  total  conventional  lesson  time  down  into  five 
components  was  of  some  interest.  To  investigate  this  question.  Model  1 was  again 
employed  as  the  full  model,  and  a restricted  model.  Model  7,  was  defined  which 
contained  a single  quantitative  measure  of  course  content  - the  total  number  of 
minutes  allocated  to  the  lesson  by  the  POI . This  variable  was  generated  by  summing 
the  values  of  the  five  component  times.  Model  7 resulted  in  an  of  .29748,  a 
reduction  from  Model  1 of  .10665,  £ (1,  2941)  = 131.597,  £ < .001.  There  would 
appear  to  be  no  question  but  that  the  use  of  the  individual  component  times  is 
well  justified. 

4. 3. 3. 3 Student  Characteristic  Measures  - The  final  set  of  comparisons  connected 
the  utility  of  the  various  AIS  pre-assessment  measures.  While  it  can  be  assumed 
that  ASVAB  scores  will  generally  be  available,  the  cost  of  obtaining  data  comparable 
to  the  pre-assessment  measures  could  only  be  justified  if  they  made  a substantial 
contribution  to  the  predictive  model's  utility.  Of  the  two  categories  of  pre- 
assessment measures,  the  affective  measures  were  considered  the  most  promising. 

It  was  reasoned  that  the  cognitive  measures  would  probably  not  add  much  to  a pre- 
dictor set  which  already  contained  ASVAB  scores.  A restricted  model.  Model  8, 

was  defined  from  which  the  four  cognitive  pre-assessment  measures  were  deleted. 

2 

Model  1 was  employed  as  the  full  model.  The  restricted  model  resulted  in  an  R 
of  .40230,  a reduction  from  Model  1 of  only  .00183.  This  difference  was  not 
significant  (£  (4,  2941)  = 2.258,  p = .061).  The  restricted  model.  Model  8,  was 
therefore  selected. 
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Finally,  the  effect  of  deleting  the  affective  variables  was  evaluated.  Model 
8 was  employed  as  the  full  model,  and  a restricted  model,  Model  9,  was  defined 
which  contained  no  pre-assessment  variables.  Model  9 generated  an  R of  .39578,  a 
reduction  from  Model  8 of  .00652,  £ (9,  2945)  = 3.570,  p < .001.  Although  the 
difference  was  significant,  the  increased  predictability  contributed  by  the  affec- 
tive pre-assessment  measures  was  not  considered  to  justify  their  cost  for  this 
application.  Therefore,  Model  9 was  retained  as  the  final  predictor  set. 

The  limited  utility  of  the  preassessment  variables  for  this  study  should  not 
be  generalized  to  their  function  as  predictors  in  an  individualized  instructional 
system.  The  preassessment  variables  selected  were  those  common  to  both  the  IM 
and  WM  courses;  however,  the  course  content  levels  sampled  by  this  study  were  much 
more  heterogeneous  than  would  generally  be  expected  in  a single,  technical  training 
course.  Course-specific,  individual  difference  measures  of  the  type  represented 
in  the  AIS  preassessment  battery  could  well  provide  the  finer  differentiation 
required  when  dealing  with  students  in  a single  course.  More  importantly,  it  must 
be  remembered  that  the  initial  CMI  mode  of  AIS  operation  from  which  this  Study's 
data  were  drawn  did  not  include  individualized  assignment  of  instructional  alterna- 
tives. If  a proposed  self-paced  course  were  to  include  alternative  instructional 
treatments  assigned  on  an  individual  basis,  measures  such  as  those  in  the  pre- 
assessment battery  could  well  make  a substantive  contribution  in  a predictive  model. 

4.3.4  Resultant  Regression  Model  to  Predict  First-Attempt-Lesson-Time  - The  19  \ 

variables  contained  in  Model  9 were  submitted  to  the  SPSS  stepwise  Multiple  Linear 
Regression  Analysis  program.  On  the  basis  of  this  analysis,  14  of  the  19  variables 
were  selected  for  the  final  regression  equation.  The  cutoff  was  made  at  the  point 
at  which  no  one  of  the  remaining  variables  signi f icantly  (£  < .01)  increased  the  ' 

overall  multiple  correlation.  The  final  14  variables,  their  coefficients  to  five 

p j 

significant  digits,  and  the  multiple  R and  R at  each  step  are  shown  in  Table  5 in  I 

their  order  of  entry  into  the  regression  equation. 

The  final  regression  equation  produced  a multiple  R of  .62832,  thus  accounting 
for  39.48  percent  of  the  variance  in  first-attempt-lesson-time  for  Sample  1.  All 
of  the  five  quantitative  course  content  measures  entered  the  equation.  As  would  be 
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TABLE  5 

MULTIPLE  LINEAR  REGRESSION  EQUATION 
PREDICTING 

FIRST  ATTEMPT  LESSON  TIME 


VARIABLE 

COEFFICIENT 

BETA  WEIGHT 

MULTIPLE  R 

Constant 

58.136 

Lecture/Discussion  Minutes 

0.11689 

0.17618 

.4886 

.2388 

Demon St rat ion/ Performance 
Minutes-Cognitive 

0.25768 

0.25651 

.5608 

.3145 

ASVAB  Mechanical  Score 

-0.26054 

-0.10415 

.5764 

.3322 

Memory  Content  Minutes 

-0.17944 

-0.19951 

.5883 

.3461 

Simple/Complex  Memory 
Rating 

14.928 

0.17171 

.6005 

. 3606 

No.  of  Self  Check  Items 

-0.98747 

-0.069196 

.6047 

.3657 

Psychomotor  Content  Minutes 

-0.20393 

-0.16736 

.6098 

.3719 

Programmed  Instruction/AV 
Minutes 

0.27739 

0.16567 

.6203 

.3848 

Sex 

-16.202 

-0.083742 

.6238 

.3891 

Demon St rat  ion/ Performance 
Minutes -Psychomotor 

-0.088128 

-0.086009 

.6255 

.3913 

ASVAB  Administrative  Score 

-0.10187 

-0.029058 

.6261 

.3919 

Simple/Complex  Psychomotor 
Rating 

1.9439 

0.056918 

.6266 

.3927 

Cognitive  Content  Minutes 

0.15746 

0.35841 

.6272 

.3934 

Outside  Assignment  Minutes 

-0.14308 

-0.28089 

.6283 

.3948 
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Strongly  related  to  the  self-paced  lesson  completion  time.  Five  of  the  six  content 
classification  measures  were  included.  It  is  interesting  to  note  that  the  measures 
Memory  Content  Minutes  and  Psychomotor  Content  Minutes  were  inversely  correlated 
with  the  criterion,  i.e.,  the  greater  the  amount  of  this  type  of  content,  the 
shorter  the  first-attempt-lesson-time.  This  implies  that  substantial  time  reduc- 
tions can  be  realized  through  self-pacing  in  these  areas.  The  one  remaining 
instructional  design  variable.  Number  of  Self-Check  Items,  also  entered  with  a neg- 
ative coefficient.  This  would  imply  that  the  time  required  for  such  post-instruc- 
tional organizers  is  more  than  justified.  Neither  of  the  two  remaining  test 
characteristic  variables.  Lesson  Test  Criterion  or  Number  of  Performance  Test  Check 
Points  was  found  to  contribute  significantly  to  the  multiple  regression  equation. 
With  regard  to  student  characteristic  variables,  sex  was  found  to  be  relevant, 
entering  with  a negative  coefficient  implying  that,  for  the  material  sampled,  males 
tend  to  progress  more  quickly  than  do  females.  Finally,  the  ASVAB  Mechanical  and 
Administrative  scores  entered  with,  as  would  be  expected,  negative  coefficients. 
Given  the  presence  of  those  two  scores,  ASVAB  General  was  not  found  to  contribute 
significantly  to  the  equation. 

4.3.5  Test  of  the  Model's  Validity  and  Generality  - The  generality  of  the 
regression  equation  model  was  evaluated  by  two  means,  cross  validation  and  a test 
of  course  independence.  The  same  constraints  which  had  been  applied  to  Sample  One 
were  applied  to  the  6849  cases  in  Sample  Two,  which,  along  with  the  casewise 
deletion  of  mission  data,  reduced  the  sample  size  to  2830  cases.  The  Sample  Two 
means  and  standard  deviations  of  the  14  predictor  variables  and  the  criterion 
variable  are  shown  in  Table  6.  The  comparable  Sample  One  values  are  also  shown 
for  the  sake  of  comparison. 

For  the  cross  validation,  the  final  regression  equation,  shown  in  Table  5,  was 

applied  to  the  Sample  Two  data.  The  resultant  distribution  of  predicted  first- 

attempt  -lesson  -times  had  a mean  of  83.91  and  a standard  deviation  of  39.23.  The 

correlation  between  the  predicted  and  actual  first-attempt-lesson-times  was  .5843, 

2 

R = .3414  with  a standard  error  of  estimate  of  50.62  minutes.  This  correlation 
was  significantly  greater  than  .000  (£  (1  , 2828)  = 1465.840,  £<  .001).  Since  the 

original  multiple  correlation  coefficient  was  .6283,  the  shrinkage  in  cross 
validation  amounted  to  .04400.  In  terms  of  R , the  loss  in  percentage  of  the 
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TABLE  6 

MEANS  AND  STANDARD  DEVIATIONS  OF  RELEVANT  VARIABLES  FROM 
SAMPLE  USED  TO  GENERATE  REGRESSION  EQUATION  (SAMPLE  ONE) 
AND  SAMPLE  USED  FOR  CROSS  VALIDATION  (SAMPLE  TWO) 


VARIABLE 

SAMPL 

: ONE 

SAMPLE  TWO 

MEAN 

STD.  DEV. 

MEAN 

STD.  DEV. 

COURSE  CONTENT  VARIABLES 

Quantitative  Measures 

Lecture/Discussion  Minutes 

89.54 

97.58 

88.00 

95.19 

Demonstrat ion/ Performance 
Minutes-Cogniti ve 

55.70 

64.45 

53.74 

63.10 

Demon  s t ra  t i on/ Perf ormance 
Minutes  - Psychomotor 

27.91 

63.19 

27.75 

62.42 

Programmed  Instruction/AV 
Minutes 

8.43 

38.67 

8.42 

38.39 

Outside  Assignment  Minutes 

51.27 

127.10 

49.65 

120.21 

Content  Classification  Measures 

Memory  Content  Minutes 

68.86 

71.99 

68.57 

72.33 

Simple/ Complex  Memory  Rating 

2.61 

0.74 

2.62 

0.75 

'O'lnitive  Content  Minutes 

145.67 

147.37 

140.26 

142.48 

F-ychomotor  Content  Minutes 

20.95 

53.13 

20.83 

51.18 

Simple/Complex  Psychomotor 
Rating 

1.23 

1.90 

1 .25 

1.93 

Instructional  Design  Variables 

No.  of  Self-Check  Items 

2.74 

4.54 

2.85 

4.65 

STUDENT  CHARACTERISTIC 

Sex  (0  = Female,  1 = Male) 

0.87 

0.33 

0.88 

0.33 

ASVAB  Scores 

ASVAB  Administrative 

59.62 

18.47 

59.93 

18.47 

ASVAB  Mechanical 

58.97 

25.88 

57.19 

22.90 

CRITERION  VARIABLE 

First  Attempt  Lesson  Time 

84.90 

64.74 

80.10 

— i 

62.36 
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variance  accounted  for  amounted  to  5.34  percent.  The  results  of  the  cross  valida- 
tion were  considered  to  be  quite  satisfactory.  It  may  be  concluded  that,  when 
applied  to  data  comparable  to  that  used  for  equation  generation,  the  regression 
equation  model  can  satisfactorily  predict  first-attempt-lesson-times. 

The  question  remained,  however,  as  to  the  extent  to  which  the  model  would 
provide  satisfactory  predictions  of  first-attempt-lesson-times  for  a new  course, 
one  containing  a different  type  of  subject  matter.  That  is,  to  what  extent  could 
the  model  be  generalized  to  courses  other  than  those  employed  in  the  data  base?  A 
thorough  evaluation  cf  the  model's  generality  would  have  required  the  construction 
of  a new  data  base  and  was,  unfortunately,  beyond  the  scope  of  the  current  study. 
However,  a partial  test  could  be  made  through  use  of  the  available  data  base. 

It  will  be  recalled  that  the  data  base  sampled  a total  of  sixty-four  lessons 
from  two  courses:  twenty-nine  from  IM,  and  thirty-five  from  WM.  The  question  of 

model  generality  was  rephrased  to  ask  to  what  extent  did  unmeasured  characteristics 
of  these  two  courses,  i.e.,  factors  specific  to  each  course,  contribute  to  the 
variability  in  first-attempt-lesson-time.  That  is,  to  what  extent  could  the 
variance  accounted  for  be  increased  by  knowledge  of  the  course  from  which  the 
lessons  and  students  were  drawn?  The  degree  to  which  knowledge  of  course  member- 
ship increased  predictability  could  be  taken  as  a rough  measure  of  the  limitation 
of  the  model  generality. 

A linear  models  comparison  was  employed  in  which  the  restricted  model.  Model  10, 
was  defined  as  containing  the  14  variables  in  the  final  regression  equation. 

A full  model.  Model  11,  was  defined  which  contained,  in  addition,  a variable 

2 

representing  course  number  (IM  = 1 , WM  = 4).  The  full  model  resulted  in  an  R of 

2 

.41300.  The  restricted  model,  the  final  regression  equation,  produced  an  R value 
of  .3948,  a reduction  of  .01820.  This  represented  a significant  reduction 
(£  (1  , 2958)  = 91.71  , £<  .001).  Thus,  specific  course  membership  accounted  for  an 
additional  2.0  percent  of  the  variance. 
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Although  significant,  the  amount  of  variance  accounted  for  by  knowledge  of 
course  membership  is  relatively  small.  Although  some  loss  in  predictability  could 
be  expected  if  the  regression  equation  model  were  applied  to  a new  course,  the 
results  of  this  final  comparison  imply  that  the  accuracy  of  prediction  would  not  be 
seriously  diminished. 

4.3.6  Generation  of  Block-Completion-Times  from  First-Attempt-Lesson-Times  - As  was 
discussed  in  paragraph  4.2.1,  Definition  of  the  Criterion  Variable,  the  criterion 
variable  of  interest  was  block-completion-time.  It  was,  however,  necessary  to 
develop  the  regression  equation  at  the  lesson  level  rather  than  at  the  block  level 
in  order  to  obtain  sufficient  variability  in  content.  Having  generated  the 
regression  equation,  it  was  now  necessary  to  provide  a transition  to  block  level 
prediction.  As  was  outlined  in  paragraph  4.2.1,  the  approach  employed  was  to 
determine  a proportional  "lost  time"  for  block  completion  and  the  sum  of  the 
completion  times  for  all  the  lessons  contained  in  the  block. 

As  was  discussed  in  paragraph  4.2.3  of  this  report,  none  of  the  nine  block 
samples  from  the  tv/o  courses  contained  a complete  set  of  lessons.  This  was  due  to 
the  fact  that  there  were  AIS  lessons  in  each  block  which  represented  new  content, 
content  for  which  no  conventional  instruction  POIs  were  available.  Since  there 
was  no  basis  for  establishing  course  content  predictor  variables  for  these  lessons, 
they  were  excluded  from  consideration.  Since  the  data  base  constructed  for  this 
study  did  not  contain  any  examples  of  complete  blocks,  it  was  necessary  to  return 
to  the  AIS  database  to  obtain  the  required  information,  i.e.,  block-completion- 
times  and  f i rst-attempt-times  for  all  of  the  lessons  in  each  block. 

Data  were  retrieved  from  each  of  the  nine  blocks  employed  in  the  current  study. 

A student's  data  were  included  in  this  analysis  only  if  reliable  first-attempt- 
lesson-times  were  available  for  all  lessons  in  the  block,  both  those  included  in 
the  original  analyses  and  those  which  had  previously  been  excluded.  Furthermore, 
a student's  data  were  excluded  if  (a)  the  recorded  block-completion-time  was  less 
than  the  sum  of  the  first-attempt-lesson-times  or  (b)  if  block-completion-time 
exceeded  reasonable  maximums  identified  by  the  AIS  materials  developers.  For  each 
student  who  met  these  constraints,  two  values  were  generated;  (a)  the  sum  of 
first-attempt-lesson  times;  and  (b)  block-completion-time.  The  difference  between 

4-35 


/VTCDO/V/Vti.*-  DOUGLAS  ASTDOIVAUTICS  COALfAh/V  - KAST 


r 


EVALUATING 
INSTRUCTIONAL  SYSTEMS 


MDC  E1570 


VOLUME  II 


15  MAY  1977 


i 


these  two  values  was  then  expressed  as  a proportion  of  the  sum  of  first-attempt-  i 

lesson-times.  This  constituted  the  "lost  time"  ratio  for  that  student.  A mean  lost  j 

time  was  then  calculated  for  each  of  these  nine  blocks.  A final  mean  was  determined  | 

on  the  basis  of  these  nine  values.  The  reason  for  computing  the  mean  in  two  steps  | 

} 

rather  than  simply  summing  across  blocks  was  to  avoid  biases  introduced  by  unequal 
numbers  of  observations  in  the  various  blocks. 

The  resultant  lost  time  ratio  was  0.40.  That  is,  on  the  average,  block 

completion  time  was  found  to  equal  1.40  x (sum  of  first-attempt-lesson-times). 

As  may  be  seen  by  reference  to  the  Descriptive  Model  (Figure  2),  this  unmeasured  l 

f 

lost  time  was  comprised  of  the  time  attributable  to  the  following  activities;  I 

i 

(a)  queuing  for  lesson  resources  and  test  scoring  (relatively  negligible  times);  |, 

(b)  remedial  study,  queuing  and  retesting  on  failed  lessons;  (c)  administrative  | 

overhead;  (d)  credential  ling  testing  and  queuing  for  test  scoring;  and  (e)  block  j. 

level  remediation  following  failure  of  a credentialling  test.  In  the  Cost  Model,  j 


to  be  discussed  in  Section  5.0  of  this  report,  the  product  of  the  regression 


evaluating 

INSTRUCTIONAL  SYSTEMS 


VOLUME  II 


MDC  El 570 
15  MAY  1977 


5.0  THE  COST  MODEL 

One  of  the  objectives  of  this  study  was  the  modification  of  a cost  analysis 
computer  program  previously  developed  by  the  Rand  Corporation.  Specifically,  the 
Rand  model  was  reprogrammed  to  use  the  regression  equation  developed  in  this  study 
as  the  method  for  generating  predictions  of  course  completion  times  for  students. 

The  unmodified  Rand  model  uses  a completion  time  entered  by  the  user  without  speci- 
fying how  this  time  was  estimated. 

The  Rand  MODIA  cost  model  (Hess  and  Kanter,  1976)  is  actually  comprised  of 
three  computer  programs;  a user  interface  program  (UI);  a resourse  utilization 
model  (RUM);  and  a cost  analysis  program  (MODCOM).  The  UI  is  an  interactive  pro- 
gram which  produces  a course  description  in  computer-compatible  data  which  can  be 
input  into  the  RUM.  The  RUM  is  a batch  program  which  produces  detailed  course 
operation  reports  regarding  student  flow  patterns  and  waiting  times  as  well  as 
resource  demand  and  utilization.  The  UI  and  RUM  are  designed  to  be  used  repeatedly 
until  the  course  planner  has  generated  several  course  designs.  The  costs  of  these 
alternative  course  descriptions  are  then  compared  by  the  batch  process  cost  model 
program  (MODCOM).  Resource  requirements  from  the  UI,  RUM,  user  supplied  course 
completion  times,  and  cost  and  manning  data  are  input  into  MODCOM  to  produce  total 
course  costs  for  up  to  five  years.  On  some  of  the  cost  and  manning  factors,  default 
values  are  used  if  not  supplied  by  the  arMlyst. 

The  computer  program  developed  in  this  study  is  named  Aid  for  Instructional 
Development  and  Evaluation  (AIDE).  It  is  a modification  of  the  MODCOM  cost 
analysis  program  and  retains  all  of  the  cost  analysis  features  and  outputs  of 
MODCOM  while  allowing  for  course  completion  time  to  be  estimated  by  the  regression 
equation  developed  in  this  study. 

Although  AIDE  inputs  are  sufficiently  documented  in  this  report  for  it  to 
be  used  by  itself,  it  is  designed  to  be  used  as  a replacement  for  MODCOM  in  the 
MODIA  series  of  programs,  and  the  detailed  MODCOM  documentation  should  be  used  with 
AIDE. 
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5.1  DESCRIPTION  AND  USE  OF  THE  AIDE  PROGRAM 

The  AIDE  program  projects  the  investment  and  operating  costs  of  a 
given  course  design  for  a five-year  time  frame.  Course  completion  time 
for  each  type  of  student  is  computed  and  used  to  calculate  student  pay 
and  allowances  costs.  The  number  of  student  entrants  per  year  and  their 
completion  times  are  used  to  determine  resource  requirements  and  costs. 

Resources  include  manpower,  courseware,  hardware,  and  facilities.  These 
resources  require  specification  in  terms  of  quantities  such  as  attrition  rates, 
and  utilization  rates.  These  specifications  are  outputs  of  the  MODIA  UI  and 
RUM  programs  or  may  be  supplied  from  knowledge  of  a specific  course.  Flexibility 
in  the  inputs  required  allows  the  model  to  be  used  at  any  level  of  detail 
desired. 

The  following  outputs  are  generated  by  the  AIDE  program: 

0 Input  Data  Listing 

0 Input  Lesson  Descriptor  Values  Listing 

0 Course  Completion  Data  Generated  by  the  Regression  Model 

0 Personnel  Distributions  and  Cost  Factors 

0 Graduate  Summary 

0 Manpower  Summary 

0 Courseware,  Hardware,  and  Facilities  Requirements  by  Year 
0 Functional  Cost  Summary. 

These  outputs  are  illustrated  in  Figures  A-1  through  A-8  in  Appendix 
A.  AIDE  outputs  4 through  8 are  identical  with  the  original  MODCOM  out- 
puts. Output  1 differs  only  in  that  the  additional  student  and  lesson 
variables  unique  to  AIDE  are  listed.  Outputs  2 and  3 are  unique  to  AIDE. 

Output  2 provides  a complete  list  of  the  input  lesson  descriptor  variables, 
and  Output  3 is  a summary  of  the  course  completion  data  generated  by  the 
regression  equation. 

A description  of  input  data  preparation  is  given  in  Appendix  A.  A 
more  detailed  description  of  the  data  input  is  provided  by  the  RAND  MODCOM 
documentation.  Any  differences  in  input  data  preparation  between  AIDE 
and  MODCOM  are  noted  specifically  in  Appendix  A,  In  general,  the  differences 
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between  the  two  programs  are  that  the  AIDE  regression  equation  requires 
input  of  values  for  three  student  descriptor  variables  for  each  type  of 
student  defined  and  values  for  the  eleven  course  descriptor  variables. 

The  particular  regression  equation  developed  in  this  study  is  currently 
hardcoded  in  the  AIDE  program,  but  AIDE  could  be  changed  to  employ  any 
similar  regression  equation  with  only  minor  changes  to  the  FORTRAN  code 
(as  noted  in  Appendix  A). 

One  limitation  of  the  AIDE  cost  analysis  program  is  that,  while  there 
is  variability  in  the  predicted  course  completion  times  generated  by  the 
regression  equation,  no  corresponding  variability  is  computed  for  course 
costs  by  the  cost  model.  r 

Although  AIDE  can  take  up  to  75  different  lessons  as>efined  by  their 
lesson  descriptor  variables,  it  should  be  noted  that  the  original  (con- 
ventional course)  lesson  lengths  in  this  study  were  approximately  one  to 
seven  hours.  Thus,  the  prediction  equation  for  individualized  les'son  and 
completion  times  should  not  be  generalized  to  lessons  whose  original 
lengths  fall  outside  this  range.  Regarding  input  data  preparation, 
original  course  lengths  outside  of  this  range  should  be  broken  down  into 
sums  of  lessons  whose  lengths  fall  within  the  above  range.' 

5.2  EXAMPLE  USE  OF  AIDE 

To  illustrate  how  the  AIDE  program  can  be  exercised,  course  completion 
and  cost  data  were  generated  for  a standard  sample  and  for  additional  samples 
representing  three  specific  questions  discussed  in  paragraphs  5.2.1  through 

5.2.3. 

Input  data  for  all  of  these  samples  were  obtained  primarily  from  example 
values  given  in  the  MODIA  documentation,  but  a number  of  values  were  repre- 
sentative of  available  AIS  data.  Specifically,  these  values  were  (a)  a 
conventional  course  length  of  50  hours,  (b)  a student  washout  rate  of  10 
percent,  (c)  a student  entry  interval  of  once  per  30  hours,  (d)  a total 
student  entry  rate  of  3200  per  year  or  approximately  60  per  week,  (e) 
lesson  descriptor  variables,  and  (f)  student  characteristic  variables. 

For  the  standard  sample,  an  approximate  normal  distribution  of  student 
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variable  values  was  chosen  for  the  ASVAB  Administrative  (ASVADM)  and 
Mechanical  (ASVAMEC)  scores.  The  mean  values  of  these  two  distributions 
were  set  equal  to  the  mean  values  of  Sample  One  from  the  regression 
analysis  section  of  this  study.  Sex  was  held  constant  by  defining  an  all 
male  sample.  Likewise,  student  personnel  type,  designation,  and  paygrade 
value  were  held  constant  for  all  students  at  values  derived  from  the 
MODIA  documentation. 

Figures  A-1  through  A-8  in  Appendix  A illustrate  the  AIDE  output  for 
this  standard  sample.  Figure  A-2  lists  the  values  of  the  lesson  variables, 
and  Figure  A-3  lists  the  values  of  the  student  variables.  The  standard 
sample  time  and  cost  results  are  summarized  in  Table  7.  The  first  portion 
of  the  table.  Course  Completion  Data,  is  derived  from  the  AIDE  program 
Output  3 (Figure  A-3  in  Appendix  A).  It  lists  the  expected  course 
completion  times  for  students  of  five  differing  ability  levels  as  measured 
by  the  ASVAB  Administrative  (ASVADM)  and  Mechanical  (ASVMEC)  scales.  The 
average  expected  course  completion  time  across  all  students  is  shown  to 
be  79.17  hours,  a savings  of  52.7  percent  from  the  conventional  course 
length  of  40  hours.  The  second  portion  of  the  table,  the  Functional  Five 
Year  Cost  Summary,  is  derived  from  AIDE  Output  8 (Figure  A-8  in  Appendix 
A).  The  figures  shown  for  each  category  are  totals  over  the  initial  five 
year  period  assuming  the  distribution  of  student  ability  levels  shown  in 
the  top  portion  of  the  table.  The  various  cost  items  are  described  on 
pages  A-27  and  -28  of  Appendix  A. 
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TABLE  7 

AIDE  TEST  RUN  FOR  STANDARD  SAMPLE 


COURSE  COMPLETION  DATA  (IN  HOURS) 


Student  Variables  Course 

Completion 

ASVADM  ASVMEC Time  (Mrs.) 

84  85  17.0 

72  72  18.1 

60  59  19.2 

48  46  20.2 

36  33  21.3 

Average  Course  Completion  Time  is  19.17  Hours 

Average  time  Savings  is  52.1  Percent 

FUNCTIONAL  FIVE  YEAR  COST  SUMMARY 
Courseware  Procurement 


Printed  Media 

$136,700 

Display  Media 

48,000 

Software 

13,100 

Hardware  Procurement 

Media  Hardware 

22,400 

Special  Equipment 

0 

Overhead  Hardware 

2,000 

Facility  Construction 

75,000 

Pay  and  Allowances 

Students 

1091 ,700 

Instructors 

1644,800 

Curriculum  Personnel 

79,300 

Facilities  Maintenance  Personnel 

204,500 

Training  Admin.  Personnel 

60,000 

Base  Operating  Support  Personnel 

335,000 

Medical  Personnel 

59,100 

PCS  Costs 

Students 

7166,700 

Instructors 

25,400 

Instructor  Training 

Factory  Training  of  Initial  Cadre 

3,000 

Education  Training 

54,200 

Miscellaneous  Operating  Costs 

Computer  Service  Charges 

330,000 

Hardware  Replenishment/Repair 

8,000 

Miscellaneous  Supplies 

43,400 

TOTAL  COURSE  COST 

$11,675,600 
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It  should  be  understood  that  the  purpose  here  was  to  illustrate  how 
the  cost  model  can  be  exercised,  not  to  provide  an  accurate  description 
of  the  cost  structure  of  an  AIS  course.  If  the  data  were  to  be  representative 
of  an  AIS  course,  it  would  be  necessary  to  increase  the  number  of  lessons 
in  the  sample  from  ten  to  about  120,  and  a number  of  additional  inputs  would 
need  to  be  modified  to  be  representative  of  either  the  IM  or  WM  course 
characteristics  and  students. 


5.2.1  Effect  of  Variation  in  Student  Aptitude  Level  - The  first  question 
investigated  asked  how  cost  would  be  affected  if  the  average  level  of 
student  aptitude  is  increased  or  decreased  where  aptitude  is  measured  by 
the  ASVAB  Administrative  and  Mechanical  scores.  The  purpose  of  this 
particular  question  was  to  represent  a situation  in  which  an  administrator 
might  wish  to  learn  the  effects  on  course  costs  of  lowering  course  entry 
requirements  (a  possible  outcome  of  zero  draft)  or  of  raising  them  (a 
possible  outcome  of  increased  unemployment  in  the  civilian  sector). 

Two  more  distributions  of  student  variables  were  generated  which 
were  also  approximately  normal  and  whose  ASVADM  and  A5VMEC  means  were 
two-thirds  of  a standard  deviation  below  and  one-half  of  a standard 
deviation  above  the  mean  of  the  standaro  sample,  respectively.  The  dis- 
tributions employed  (including  the  standard  sample)  are  illustrated  in 
Table  8.  For  each  sample,  the  same  number  of  students  was  input  for  each 
of  five  years,  and  all  other  input  variables  were  held  constant.  With  an 
entry  interval  of  one  week  (30  hours),  the  entry  rate  was  64  students  per 
week,  roughly  the  number  of  weekly  student  entries  in  the  AIS  IM  course  at 
Lowry  AFB. 
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TABLE  8 

DISTRIBUTIONS  OF  STUDENT  VARIABLES  FOR  COMPARISON  OF 
STANDARD,  BELOW  AVERAGE,  AND  ABOVE  AVERAGE  STUDENT  POPULATIONS 


SAMPLE  DESCRIPTION 

ASVADM 

ASVMEC 

STUDENT  ENTRANTS  PER  YEAR 

84 

85 

200 

72 

72 

800 

Standard  Sample 

60 

59 

1200 

48 

46 

800 

36 

33 

200 

96 

98 

200 

84 

85 

800 

Above  Average  Sample 

72 

72 

1200 

60 

59 

800 

48 

46 

200 

72 

72 

200 

60 

59 

800 

Below  Average  Sample 

48 

46 

1200 

36 

33 

800 

24 

20 

200 

The  time  and  cost  results  for  the  two  Above  and  Below  Average  student 
samples  are  given  in  Tables  9-1  and  9-2,  respectively.  These  Tables  are  in 
the  same  format  as  Table  7,  and  the  data  were  derived  from  the  same  AIDE 
program  outputs,  i.e..  Output  3 and  Output  8. 

For  purposes  of  comparison,  data  from  Tables  7,  9-1  and  9-2  are  summarized 
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TABLE  9-1 


AIDE  TEST  RUN  FOR  ABOVE  AVERAGE  STUDENT  SAMPLE 


COURSE  COMPLETION  DATA 

(IN  HOURS) 

STUDENT 

ASVADM 

VARIABLES 

ASVMEC 

COURSE 
COMPLETION 
TIME  (HRS.) 

96 

85 

16.7 

84 

72 

17.8 

72 

59 

18.9 

60 

46 

20.0 

48 

33 

21 .0 

Average  Course  Completion  Time  is  18,88  Hours 
Average  Time  Savings  is  52.8  Percent 


FUNCTIONAL  FIVE  YEAR  COST  SUMMARY 


Courseware  Procurement 


Printed  Media 

$ 136,700 

Display  Media 

48,000 

Software 

13,100 

Hardware  Procurement 

Media  Hardware 

20,900 

Special  Equipment 

0 

Overhead  Hardware 

1 ,800 

Facility  Construction 

75,000 

Pay  and  Allowances 

Students 

1075,700 

Instructors 

1644,800 

Curriculum  Personnel 

79,300 

Hardware  Maintenance  Personnel 

195,000 

Facilities  Maintenance  Personnel 

60,000 

Training  Admin.  Personnel 

335,000 

Base  Operating  Support  Personnel 

273,300 

Medical  Personnel 

59,100 

PCS  Costs 

Students 

7166,700 

Instructors 

25,400 

Instructor  Training 

Factory  Training  of  Initial  Cadre 

3,000 

Education  Training 

54,200 

Miscellaneous  Operating  Costs 

Computer  Service  Charges 

330,000 

Hardware  Replenishment/Repair 

7,500 

Miscellaneous  Supplies 

42,900 

TOTAL  COURSE  COST 

$11,647,400 
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TABLE  9-2 

AIDE  TEST  RUN  FOR  BELOW  AVERAGE  STUDENT  SAMPLE 

COURSE  COMPLETION  DATA  (IN  HOURS) 

STUDENT  VARIABLES  COURSE 

COMPLETION 

ASVADM  ASVMEC  TIME  (HRS.) 


72 

85 

17.3 

60 

72 

18.4 

48 

59 

19.5 

36 

46 

20.5 

24 

33 

21.6 

Average  Course  Completion  Time  is  19.45  Hours 
Average  Time  Saving  is  51.4  Percent 

FUNCTIONAL  FIVE  YEAR  COST  SUMMARY 


Courseware  Procurement 


Printed  Media 

$ 136,700 

Display  Media 

48,000 

Software 

13,100 

Hardware  Procurement 

Media  Hardware 

22,400 

Special  Equipment 

0 

Overhead  Hardware 

2,000 

Facility  Construction 

• 75,000 

Pay  and  Allowances 

Students 

1108,100 

Instructors 

1644,800 

Curriculum  Personnel 

79,300 

Hardware  Maintenance  Personnel 

204,500 

Facilities  Maintenance  Personnel 

60,000 

Training  Admin.  Personnel 

335,000 

Base  Operating  Support  Personnel 

278,300 

Medical  Personnel 

59,100 

PCS  Costs 

Students 

7166,700 

Instructors 

25,400 

Instructor  Training 

Factory  Training  of  Initial  Cadre 

3,000 

Education  Training 

54,200 

Miscellaneous  Operating  Costs 

Computer  Service  Charges 

330,000 

Hardware  Replenishment/Repai r 

8,000 

Miscellaneous  Supplies 

43,900 

TOTAL  COURSE  COST 

$11,697,500 
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in  Table  10.  As  may  be  seen  from  Table  10,  dramatic  variations  in  aptitude 
in  the  student  population,  as  measured  by  the  ASVAB  Administrative  and 
Mechanical  scores,  have  relatively  little  effect  on  course  completion  time, 
approximately  1.5  percent.  This  is  due  to  the  fact  that,  in  the 
regression  equation  employed  (see  Table  5),  these  two  variables,  although 
significant,  account  for  relatively  little  of  the  variability  in  first- 
attempt-lesson-time.  The  effect  on  Total  Course  Cost  is  even  less, 
approximately  0.2  percent.  This  is  due  to  the  relatively  large  fraction 
of  the  course  costs  which  are  considered  by  the  model  to  be  fixed  rather 
than  dependent  on  variable  course  length. 

TABLE  10 

TIME  AND  COST  COMPARISONS  FOR  DIFFERENT  STUDENT  DISTRIBUTIONS 


SAMPLE  DESCRIPTION 

AVERAGE  TIME 

STUDENT 

PAY  COST 

TOTAL  COURSE  COST 

Hours 

$* 

■S* 

Above  Average 

B 

1075.7 

-1.5 

11647.4 

-0.24 

Standard 

MM 

1091.7 

0.0 

11675.6 

0.00 

Below  Average 
1 - 

m 

1.4 

1108.1 

1.5 

J 

11697.5 
1 

0.19 

* In  Thousands  of  Dollars 


5.2.2  Effect  of  Variation  of  Proportion  of  Memory  Versus  Cognitive  Content  - 
The  second  question  investigated  the  effect  of  variations  in  the  relative 
proportions  of  Memory  to  Cognitive  content.  The  purpose  of  this  question 
was  to  illustrate  the  differing  outcomes  that  may  be  expected  when  courses 
with  different  characteristics  are  individualized.  The  variables  in  the 
regression  equation  (Table  5)  indicated  that  individualization  of  a course 
containing  a relatively  large  amount  of  memory  content  would  result  in 
more  substantive  time  savings  than  would  individualization  of  an  other- 
wise similar  course  with  less  memory  and  more  cognitive  content. 
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To  investigate  this  question,  the  number  of  minutes  of  memory  content 
material  (MEM)  in  the  course  represented  in  the  standard  sample  was 
increased  by  one  standard  deviation,  i.e.,  72  minutes  per  lesson.  In 
order  to  keep  lesson  lengths  constant,  a corresponding  reduction  of  72 
minutes  per  lesson  was  made  in  the  amount  of  cognitive  content  (COG).  This 
represented  a reduction  of  0.49  standard  deviations  in  the  cognitive  content. 

The  Time  and  Cost  results  for  this  high-memory-content  course 
are  given  in  Table  11  which  employs  the  same  format  as  Table  7.  For 
purposes  of  comparison  with  the  standard  sample,  data  from  Tables  7 and  11 
are  summarized  in  Table  12.  As  may  be  seen  from  Table  12,  substantially 
greater  time  savings  would  be  expected  from  individualization  of  a course 
with  a higher  percentage  of  memory  content.  While  the  predicted  average 
course  completion  time  for  the  standard  sample  course  was  19.17  hours 
(down  from  40  hours  as  taught  by  conventional  methods),  the  predicted 
average  course  completion  time  for  the  high-memory-content  course  (also 
assuming  a 40  hour  conventional  course  baseline)  was  only  13.51  hours. 

Again,  it  should  be  noted  that  while  there  are  large  differences 
in  student  pay  costs  between  the  two  types  of  courses  (29.5  percent),  the 
difference  in  Total  Course  Cost  is  relatively  small  (8.1  percent). 
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TABLE  n 

AIDE  TEST  RUN  FOR  HIGH-MEMORY/LOW-COGNITIVE-CONTENT  COURSE 
COURSE  COMPLETION  DATA  (IN  HOURS) 


STUDENT 

ASVADM 

VARIABLES 

ASVMEC 

COURSE 
COMPLETION 
TIME  (HRS.) 

84 

85 

11.4 

72 

72 

12.4 

60 

59 

13.5 

48 

46 

14.6 

36 

33 

15,7 

Average  Course  Completion  Time  Is  13,51  Hours 
Average  Time  Saving  Is  66.2  Percent 


FUNCTIONAL  FIVE  YEAR  COST  SUMMARY 


Courseware  Procurement 


Printed  Media 

$ 136,600 

Display  Media 

48,000 

Software 

13,100 

Hardware  Procurement 

Media  Hardware 

13,400 

Special  Equipment 

0 

Overhead  Hardware 

- 300 

Facility  Construction 

75,000 

Pay  and  Allowances 

Students 

769,300 

Instructors 

1252,800 

Curriculum  Personnel 

79,300 

Hardware  Maintenance  Personnel 

142,500 

Facilities  Maintenance  Personnel 

60,000 

Training  Admin.  Personnel 

289,500 

Base  Operating  Support  Personnel 

205,300 

Medical  Personnel 

43,100 

PCS  Costs 

Students 

7166,700 

Instructors 

19,200 

Instructor  Training 

Factory  Training  of  Initial  Cadre 

3,000 

Education  Training 

40,700 

Miscellaneous  Operating  Costs 

Computer  Service  Charge 

330,000 

Hardware  Replenishment/Repair 

5,500 

Miscellaneous  Supplies 

32,400 

TOTAL  COURSE  COST 

$10,725,100 

5-12 


MCOOMMCLL  OOUaCJKS  ASTHOM/tUTiCS  COI^fArvV  • MAmr 


MDC  El 570 
15  MAY  1977 


r 


EVALUATING 

INSTRUCTIONAL  SYSTEMS  VOLUME  II 


MDC  El 570 
15  MAY  1977 


TABLE  12 

TIME  AND  COST  COMPARISONS  FOR  STANDARD  AND  HIGH  MEMORY/LOW  COGNITIVE 

CONTENT  COURSES 


SAMPLE  DESCRIPTION 

AVERAGE  TIME 

STUDENT 

PAY  COST 

J 

TOTAL  COURSE  C0ST| 

Hours 

% Change 

S* 

% Change 

$* 

7o  Change 

Standard  Sample  with 
Mean  Values  on  MEM, 
COG 

19.17 

0.0 

1091.7 

0.0 

11675.6 

0.0 

Standard  Sample  with 
MEM  increased  1 S.D. , 
COG  decreased  0.49  S.D. 

13.51 

-29.5 

769.3 

-29.5 

10725.1 

-8.1 

* In  Thousands  of  Dollars 


5.2.3  Effect  of  Variation  of  Conventional  Course  Presentation  Methods  - One 
of  the  most  striking  aspects  of  the  regression  equation  to  predict  first- 
attempt-lesson-time  (Table  5)  was  the  difference  between  the  coefficients 
of  two  of  the  quantitative  course  content  variables:  Lecture/Discussion 

Minutes  and  Demonstration/Performance  Minutes  - Cognitive.  According  to 
these  coefficients,  each  additional  minute  of  Lecture/Discussion 
presentation  in  the  conventional  course  would  be  expected  to  increase 
first  attempt  lesson  time  in  the  individualized  course  by  only  0.12 
minutes.  In  contrast,  each  additional  minute  of  Demonstration/ 
Performance-Cognitive  presentation  would  be  expected  to  increase  first- 
attempt-lesson-time  in  the  individualized  course  by  0.26  minutes.  On 
the  assumption  that  these  values  were  approximately  correct,  a final 
comparison  was  made  to  illustrate  the  differing  effects  of  individual- 
izing two  courses  which  differed  on  these  dimensions. 

For  the  purpose  of  this  comparison,  the  number  of  minutes  of 
Cognitive  Demonstration  Performance  time  (CDPMC)  was  increased  by  0.75 
standard  deviations  (48  minutes)  per  lesson.  A corresponding  reduction 
of  48  minutes  per  lesson  in  Lecture/Demonstration  time  (CLOM)  amounted 
to  a change  of  0.49  standard  deviations. 
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The  time  and  cost  results  for  this  high-demonstration/performance- 
method  course  are  given  in  Table  13  which  employs  the  same  format  as  Table 
7.  For  purposes  of  comparison  with  the  standard  sample,  data  from  Tables  7 
and  13  are  summarized  in  Table  14.  As  may  be  seen  from  Table  14,  no  time 
savings  would  be  expected  from  the  individualization  of  a course  which 
employed  Demonst rat ion/ Performance  methods  for  this  very  high  proportion 
of  the  content.  The  predicted  average  course  completion  time  for  the 
individualized  high-demonstration/performance-method  course  was  44.52  hours, 
an  increase  of  4.52  hours  over  the  conventional  course. 
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TABLE  13 

AIDE  TEST  RUN  FOR  HIGH-DEMONSTRATION/PERFORMANCE-METHOD  COURSE 


COURSE  COMPLETION  DATA  (IN  HOURS) 


STUDENT 

ASVADM 

VARIABLES 

ASVMEC 

COURSE 
COMPLETION 
TIME  (HRS.) 

84 

85 

42.4 

72 

72 

43.4 

60 

59 

44.5 

48 

46 

45.6 

36 

33 

46.7 

Average  Course  Completion  Time 

Is  44.52  Hours 

Average  Time  Saving  is  -11.3  Percent 


FUNCTIONAL  FIVE  YEAR  COST  SUMMARY 


Courseware  Procurement 


Printed  Media 
Display  Media 
Software 

$ 137,800 
48,000 
13,100 

Hardware  Procurement 

Media  Hardware 
Special  Equipment 
Overhead  Hardware 

51,400 

0 

Facility  Construction 
Pay  and  Allowanc-^s 

150,')i;u 

Students 

Instructors 

Curriculum  Personnel 

Hardware  Maintenence  Personnel 

Facilities  Maintenance  Personnel 

Training  Admin.  Personnel 

Base  Operating  Support  Personnel 

Medical  Personnel 

2531 ,600 
3751,700 
79,300 
463,600 
115,500 
544,300 
607,400 
131  ,500 

PCS  Costs 

Students 

Instructors 

7154,300 

67,400 

Instructor  Training 

Factory  Training  of  Initial  Cadre 
Education  Training 

3,000 

125,400 

Miscellaneous  Operating  Costs 

Computer  Service  Charges 
Hardware  Replenishment/Repair 
Miscellaneous  Supplies 

410,000 

17,500 

95,600 

TOTAL  COURSE  COST 

$16,511,200 
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TABLE  14 

TIME  AND  COST  COMPARISONS  FOR  STANDARD  AND  HIGH-DEMONSTRATION/ 
PERFORMANCE-METHOD  COURSES 


SAMPLE  DESCRIPTION 

AVERAGE  TIME 

STUDENT  PAY  COST 

TOTAL  COURSE  COST 

Standard  Sample  with 
Mean  Values  on 
CDPMC,  CLDM 

Hours 

% Change 

$* 

% Change 

$* 

7c  Change 

19.17 

0.0 

1091.7 

n.o 

11675.6 

0.0 

Standard  Sample  with 
CDPMC  increased  .75 
S.D.  CLDM  decreased 
.49  S.D. 

44.52 

132.2 

2531.6 

131 .9 

16511.2 

41.4 

* In  Thousands  of  Dollars 


5.3  DISCUSSION  OF  AIDE  EXAMPLE  RUN  RESULTS 

The  three  questions  discussed  above  represent  simple  examples  of  how 
the  AIDE  cost  model  analysis  program  may  be  exercised,  using  differing 
student  and  lesson  variable  values,  to  investigate  the  expected  cost 
savings  resulting  from  individualizing  different  types  of  courses.  Again, 
it  should  be  emphasized  that  the  data  presented  were  not  intended  to  be 
representative  oi  the  AIS  courses  from  which  the  regression  equation  data 
base  was  derived.  Neither  was  the  intent  to  exploit  fully  the  capabilities 
of  the  Rand  MODCOM  program  from  which  the  AIDE  program  was  derived. 

There  are  a number  of  general  points  which  should  be  explicated. 

The  various  example  runs  indicate  that  direct  student  costs,  pay  and  allow- 
ances, account  for  relatively  little  of  the  total  course  costs.  For  the 
Standard  Sample  (Table  7),  student  pay  and  allowances  account  for  only 
9.35  percent  of  the  total.  Although,  as  is  discussed  below,  some  of  the 
cost  figures  provided  are  somewhat  misleading,  these  values  do  illustrate 
the  point  that  military  technical  training  does  have  substantial  fixed 
costs  which  are  relatively  insensitive  to  reductions  in  training  time 
per  se. 
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These  additional  costs  were,  however,  inflated  by  two  pre-analysis 
decisions  reflected  in  the  input  data.  First,  the  input  data  indicated 
that  student  relocation  for  training  was  to  be  treated  as  a permanent  change 
of  station  (PCS).  Given  the  predicted  average  course  length  of  19.17  hours 
(Standard  Sample,  Table  7),  it  would  be  more  cost-effective  to  handle 
student  relocation  through  temporary  assignment  - TOY.  PCS  costs  for  the 
standard  sample  were  $7,166,700  over  the  five  year  period;  six  and  one-half 
times  as  much  as  student  pay  and  allowances  over  the  same  period.  If 
relocation  for  training  had  been  handled  by  means  of  TOY  assignment  and 
if  one  assumed  a six  hour  training  day,  this  expense  would  have  been  reduced 
to  approximately  $2,903,500.  This  figure  still  represents  over  two  and 
one-half  times  the  cost  of  student  pay  and  allowances. 

A second  factor  which  increased  predicted  dosts  was  the  decision  that 
the  entry  interval  should  be  30  class  hours  or  once  per  week.  Given  that 
the  average  course  completion  time  was  predicted  to  be  less  than  20  hours, 
very  few  students  would  still  be  present  in  the  course  at  the  end  of  the 
week.  While  the  average  student  load  was  computed  to  be  36.2,  the  actual 
number  would  range  from  a high  of  at  least  64  students  at  the  time  of  a 
class  entry  to  a low  of  near  zero  just  prior  to  the  next  class  entry.  This 
represents  an  extremely  inefficient  use  of  both  personnel  and  facilities. 
Although  such  a test  run  was  not  made,  it  can  be  anticipated  that  a 
substantive  cost  reduction  would  result  if  the  entry  interval  was  reduced 
to  six  hours,  allowing  12  to  13  new  students  to  enter  the  course  each  day. 

Although  these  two  factors,  permanent  change  of  station  and  once  per 
week  entry,  did  inflate  the  estimated  costs  relative  to  student  pay  and 
allowances,  the  conclusions  drawn  concerning  the  relati ve  costs  of  the 
various  samples  run,  i.e.,  above  and  below  average  student  ability  levels, 
high-memory/low-cognitive  content,  etc.,  remain  valid.  Even  with  TOY 
assignment  and  daily  course  entry,  student  pay  and  allowances  would  still 
contribute  only  a fairly  small  proportion  to  the  total  course  costs. 
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' 6.0  CONCLUSIONS  AND  RECOMMENDATIONS 


This  study  represents  the  first  effort  to  employ  data  from  an  operational 
environment  in  an  attempt  to  relate  variables  such  as  course  content,  instructional 
and  system  design  considerations,  and  student  characteristics  to  completion  time  in 
a self-paced  course  and  to  the  cost  of  developing,  implementing,  and  maintaining 
such  a course.  The  study  has,  we  believe,  demonstrated  that  a viable  time  savings 
and  cost  model  can  be  developed  given  a sufficient  data  base  from  which  to  draw 
information.  The  generality  and  precision  of  such  a model  is,  however,  dependent 
on  the  depth  to  which  the  instructional  system  from  which  the  data  are  derived 
can  be  fully  and  explicitly  described.  Analogously,  the  utility  of  the  model  for 
a given  application  will  depend  on  the  accuracy  with  which  the  course  being  con- 
sidered for  self-pacing  is  described.  In  addition  to  a description  of  the  course 
content,  the  model  requires  a number  of  assumptions  concerning  instructional  and 
system  design  variables  and  the  characteristics  of  the  students  to  be  enrolled  in 
the  course.  The  predictive  accuracy  of  the  model  will,  of  course,  always  be  limited 
by  the  validity  of  these  assumptions. 

As  was  stated  above,  the  authors  believe  that  the  current  study  has  demon- 
strated the  feasibility  of  developing  time  savings  and  cost  models  which  have 
practical  utility.  It  must  be  recognized,  however,  that  there  is  a number  of 
limitations  both  to  the  specific  model  developed  here  and  to  future  efforts  which 
employ  this  work  as  a starting  point.  Each  of  the  three  major  components  of  the 
overall  effort,  the  descriptive  model,  the  predictive  model,  and  the  cost  model, 
will  be  discussed  with  respect  to  these  limitations  as  well  as  with  respect  to 
their  positive  aspects. 

The  descriptive  model  of  computer-based  training  systems  can  be  viewed  as 
serving  as  a framework  for  the  predictive  model  and  a guide  to  its  development. 

The  utility  of  the  predictive  model  will  always  be  limited  by  the  generality  and 
completeness  of  the  descriptive  model.  While  the  descriptive  model  employed  in 
this  study  would  appear  to  be  appropriate,  it  must  be  recognized  that  its  adequacy 
was  not  actually  tested  by  the  current  effort, since  the  predictive  model  developed 
directly  addressed  only  a small  portion  of  the  total  descriptive  model. 
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It  is  suggested,  however,  that  the  general  characteristics  of  the  descriptive 
model  proposed  are  appropriate  to  an  effort  of  this  type,  i.e.,  description  of  an 
instructional  system  in  terms  of  a number  of  time-consuming  components  and  the 
probabilities  associated  with  a limited  number  of  decision  points.  Our  experience 
with  predicting  a single  time  component  (first-attempt-lesson-time)  indicates  that 
it  is  feasible  to  develop  miniature  predictive  models  for  the  remaining  components, 
given  the  availability  of  reasonably  accurate  data. 

The  one  exception  to  this  generally  optimistic  view  must  be  noted  - prediction 
of  the  time  component  entitled  Cumulative  Administrative  Overhead.  The  quantity  of 
administrati ve  overhead  would  appear  to  be  heavily  dependent  on  local  course 
policies.  While  this  assumption  needs  to  be  examined  in  greater  detail,  it  would 
appear  that  the  time  attributable  to  this  component  may  be  too  specific  to  local 
situations  to  be  estimated  with  any  degree  of  accuracy  on  the  basis  of  available 
data. 

A second  problem  area  lies  in  estimating  the  probabilities  associated  with  the 
pass/fail  decisions  following  the  various  lesson  and  unit  tests.  As  was  discussed 
in  Paragraph  4.2.1  of  this  report,  these  decisions  again  appear  to  be  highly  dependent 
on  local  course  policies.  Consequently,  predictions  based  on  data  from  any  given 
course  or  set  of  courses  will  be  biased  by  the  policies  in  effect  in  those  courses 
and  fail  to  generalize  to  a new  course.  The  best  solution  to  this  problem  may  be  to 
establish  tolerable  failure  rates  as  part  of  the  instructional  system  design  speci- 
fications and  assume  that  instructional  materials  and  procedures  will  be  developed 
such  that  stuccnt  performance  will  meet  these  specifications. 

It  should  be  noted  that  the  descriptive  model  can  be  viewed  as  consisting  of 
several  nested  loops:  the  time  required  to  complete  the  major  instructional  unit; 

the  time  required  to  complete  a given  lesson;  and  the  time  required  by  the  indi- 
vidual components  within  the  lesson  completion  loop.  As  a result,  the  model  has  the 
interesting  attribute  that  it  is  not  necessary  that  all  of  the  individual  components 
be  modeled  explicitly.  To  the  extent  that  individual  components  are  modeled 
accurately,  the  overall  estimate  of  instructional  unit  completion  time  becomes  more 
responsive  to  variations  in  content,  student  characteristics,  and  instructional 
method.  The  model  still  has  utility,  however,  if  only  gross  estimates  are  made  of 
the  total  time  required  by  the  set  of  components  which  were  not  modeled.  The 
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initial  effort  represented  by  this  study  exemplifies  the  extreme  case  of  such  an 
approach.  The  single  component,  f i rst-attempt-1 esson-time , was  modeled  as  accu- 
rately as  possible  under  the  circumstances.  The  cumulative  time  attributable  to 
the  remaining  components  was  then  estimated  as  a function  of  first-attempt-lesson- 
time.  The  final  result,  although  not  as  accurate  as  might  be  desired,  does  provide 
a useful  estimate  of  the  total  time  required  to  complete  an  arbitrary  instructional 
unit.  Detailed  modeling  of  any  additional  component  will  increase  the  accuracy  of 
the  overall  estimate.  Thus,  further  refinement  of  the  model  can  proceed  step-by- 
step  as  more  data  become  available. 

Discussion  of  the  predictive  model  brings  us  to  the  actual  mechanics  involved 
in  estimating  the  time  required  for  an  individual  student  to  complete  an  arbitrary 
instructional  unit.  It  is  of  some  interest  to  compare  the  methodology  and  results 
of  the  current  study  with  the  work  reported  by  Wagner,  Behringer  and  Pattie  (1973) 
in  predicting  individualized  course  completion.  The  major  differences  between  the 
two  efforts  are  a function  of  their  differing  objectives.  While  Wagner,  et  al, 
were  interested  in  predicting  the  time  required  to  complete  a particular  self-paced 
course,  the  intent  of  the  work  reported  here  was  to  predict  the  time  required  to 
complete  an  arbitrary  instructional  unit,  given  a limited  set  of  characteristics  of 
that  unit.  Specifically,  whereas  all  of  the  students  in  the  Wagner,  et  al,  study 
worked  through  the  same  set  of  materials,  the  criterion  variable  in  the  current 
study  represented  the  time  to  complete  each  of  66  different  instructional  units. 

The  fact  that  the  multiple  correlation  coefficient  of  .6283  obtained  here  compares 
favorably  with  the  correlations  of  .65  to  .75  reported  by  Wagner,  et  al , indicates 
I that  the  lesson  descriptor  variables  employed  in  the  current  study  did  accurately 

represent  the  characteristics  of  the  various  lessons.  While  Wagner,  et  al , were 
able  to  obtain  substantially  higher  correlations  (.85  to  .87)  through  the  use  of 
wi thin-course  measures,  such  measures  would  not  have  been  appropriate  to  the  objec- 
tives of  the  current  study.  Wagner,  et  al , also  found  that  the  measurement  of 
individual  difference  characteristics  through  the  use  of  course  content-related 
instruments  was  preferable  to  the  use  of  the  standard  military  battery.  The 
current  study  found  that  the  use  of  available  preassessment  measures  did  not  appre- 
ciably increase  the  accuracy  of  prediction  in  the  presence  of  ASVAB  scores,  but  the 
results  might  have  been  different  had  the  preassessment  battery  contained  more 
content-specific  tasks.  Again,  however,  such  content-specific  measures  would  have 
very  limited  utility  given  the  objectives  of  the  current  work. 

6-3 


/vrcoo/v/vfl.1.  DOUGLAS  ASTHOIS/AUTICS  COIVtF‘AIS/Y  - BAST 


EVALUATING 
INSTRUCTIONAL  SYSTEMS 


VOLUME  II 


MDC  El 570 
15  MAY  1977 


The  regression  analysis  approach  employed  in  this  study  would  also  appear  to 
be  appropriate  for  estimating  the  time  required  for  many  of  the  remaining  components 
of  the  Descriptive  Model  such  as  Remedial  Study  Time  on  Lesson  i and  the  time 
required  for  the  various  diagnostic  and  credentuall ing  tests.  In  other  cases,  such 
as  estimating  the  time  consumed  in  obtaining  resources  and  test  scoring,  an  alter- 
nate approach  based  on  queuing  theory  might  well  be  more  appropriate. 

It  should  be  noted  that  the  current  study  did  not  fully  investigate  all  aspects 
of  the  regression  model  approach.  Only  simple,  linear  predictors  were  employed. 

There  is  reason  to  believe  that  accuracy  of  prediction  could  have  been  improved  if 
the  effects  of  nonlinear  transformations  of  the  predictor  variables  had  been  inves- 
tigated. For  example,  it  is  reasonable  to  expect  that  the  relationship  between 
first-attempt-lesson-time  and  student  individual  differences  could  be  better  repre- 
sented by  a curvilinear  relationship  than  by  a simple  linear  relationship.  The 
failure  to  investigate  interactions  among  the  various  predictor  variables  represents 
an  even  more  serious  limitation.  The  final  regression  model  does  not  include  any 
terms  which  are  products  of  two  or  more  predictor  variables.  As  a result,  for  exam- 
ple, the  final  regression  model  contains  the  assumption  that  while  two  students  of 
differing  ability  levels  will  complete  a given  lesson  in  differing  amounts  of  time, 
the  difference  between  their  two  completion  times  will  be  independent  of  the  amount 
of  lesson  content.  That  is,  if  Student  A requires  ten  more  minutes  than  does 
Student  B to  complete  a particular  lesson,  the  model  implies  that  Student  A will 
also  require  only  ten  more  minutes  than  Student  B to  complete  a second  lesson  con- 
taining twice  as  much  content  as  the  first  lesson.  The  incongruity  would  be  resolved 
if  the  predictor  set  contained  one  or  more  variables  which  were  the  multiplicative 
products  of  appropriate  student  ability  and  course  content  measures.  The  same  line 
of  argument  implies  the  desirability  of  including  product  terms  which  incorporate 
instructional  design  variables  such  as  the  number  of  self-check  items. 

The  effort  required  for  adequate  investigation  of  potential  transformations  and 
product  terms  was  simply  not  feasible  within  the  temporal  and  fiscal  constraints  of 
the  current  study.  Investigation  of  at  least  interaction  terms  should  certainly  be 
given  consideration  in  any  work  directed  at  refining  the  predictive  model. 


The  major  limitations  of  the  current  study  can  be  traced  directly  to  the  lack  of 
a sufficiently  broad  and  reliable  data  base.  To  a great  extent,  any  future  efforts 
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in  this  area  will  be  limited  by  these  same  constraints.  In  general,  data  acquisition 
problems  fall  into  four  broad  categories:  self-paced  student  performance  data;  base- 

line course  content  descriptions;  the  variability  of  examples  of  instructional 
methods;  and  the  distribution  of  observations  across  variations  in  course  content, 
instructional  methods,  and  student  individual  difference  characteristics. 

At  the  time  the  data  collection  effort  for  the  current  study  was  being  con- 
ducted, the  AIS  was  in  a relatively  rudimentary  stage  of  development,  and  the  only 
reliable  lesson  level  time  data  available  were  fi rst-attempt-lesson-times.  There- 
fore, the  predictive  model  was  limited  in  explicit  prediction  to  this  single 
component.  Since  that  time,  further  development  of  the  AIS  has  allowed  collection 
of  reliable,  remedial  study  times  and  lesson  completion  times.  Were  the  study  to 
be  repeated  now,  a predictive  model  could  be  developed  which  addressed  several  com- 
ponents of  the  instructional  process.  Obviously,  future  efforts  will  be  more 
fruitful  if  they  are  conducted  in  the  context  of  an  instructional  system  with  a 
fully  developed  data  collection  and  recording  component. 

While  there  are  advantages  to  conducting  future  modeling  efforts  in  the  context 
of  an  established  system,  there  is  a definite  disadvantage  in  attempting  to  model 
courses  which  have  been  conducted  under  the  auspices  of  such  a system  for  any  length 
of  time.  The  modeling  approach  employed  in  this  study  assumes  the  existence  of 
course  planning  documents  which  pertain  to  the  course  as  it  was  taught  in  a conven- 
tional, group-paced  mode.  Since  technical  training  courses  do  change,  there  is, 
with  time,  less  and  less  overlap  between  the  current  content  of  the  self-paced  course 
and  its  content  when  it  was  taught  in  the  conventional  mode.  It  will  be  recalled 
that  none  of  the  nine  blocks  of  instruction  sampled  in  this  study  contained  exactly 
the  same  content  d's  they  did  in  the  earlier,  conventional  course.  If  the  study  were 
to  be  repeated  now,  there  would  be  even  less  overlap.  Proposed  changes  to  the 
Inventory  Management  course  indicate  that  within  a year  almost  none  of  the  content 
in  the  original,  group-paced  course  will  still  be  taught.  It  is  therefore  suggested 
that  any  future  efforts  at  model  refinement  be  conducted  in  the  context  of  an  estab- 
lished system  to  which  new  courses  are  being  added. 

There  is  undoubtedly  a tendency  for  a given  instructional  system  to  shape  the 
instructional  methods  employed  in  that  system.  One  tends  to  find  relatively  little 
variability  in  method  across  lessons.  Thus,  the  effect  of  advanced  organizers  such 
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as  the  presentation  of  lesson  objectives  could  not  be  evaluated  in  the  current  study 
since  all  of  the  lessons  sampled  contained  such  objectives.  Similarly,  the  numbers 
of  embedded  questions  and  lesson  test  items  were  found  to  be  so  highly  correlated 
with  the  amount  of  lesson  content  that  it  was  necessary  to  exclude  them  from  the 
predictor  set.  Relatively  little  of  their  variability  was  independent  of  lesson 
length.  While  the  post-instructional  organizer.  Number  of  Self- Check  Items,  was 
retained  in  the  predictor  set,  the  typical  number  of  such  items  per  lesson  was  so 
small  that  one  should  be  hesitant  in  using  the  regression  equation  information  to 
evaluate  the  utility  of  such  organizers.  It  would  be  desirable  to  sample  a much 
wider  variety  of  values  of  course  content  variables  in  any  future  efforts  at  model 
refinement.  The  current  study's  limitation  to  two  courses  makes  it  difficult  to 
disentangle  the  effects  due  to  differences  in  course  content  from  those  due  to 
unmeasured,  course-specific  factors.  Finally,  it  would  be  highly  desirable  if 
future  work  could  include  examination  of  a variety  of  alternative  instructional 
media,  including  computer-assisted  instruction. 


The  final  data  problem  area  concerns  the  intercorrelation  of  predictor  variables. 
If  it  were  possible  to  manipulate  directly  the  variables  of  interest,  one  would 
impose  experimental  controls  to  hold  other  conditions  constant  while  varying  one,  or 
at  most  a limited  set,  of  variables.  Since  this  was  not  possible,  the  current  study 
employed  a regression  analysis  approach  in  an  attempt  to  examine  statistically  the 
effects  of  individual  variables.  The  regression  approach  is  limited,  however,  in 
that  the  confounding  effects  due  to  the  intercorrelations  among  variables  cannot  be 
i controlled.  For  example,  if  a particular  instructional  method  was  always  employed 

' for  teaching  a particular  class  of  content,  and  never  employed  for  teaching  other 

categories  of  content,  its  effect  could  never  be  evaluated.  Nor  could  one  determine 
the  expected  time  savings  due  to  self-pacing  with  other  instructional  methods  for 
that  class  of  content.  Other,  less  extreme  examples  can  also  present  problems. 

Within  the  current  AIS  system,  the  more  expensive  audio-visual  media  are  generally 
employed  only  for  those  lessons  which  experience  has  proven  to  be  particularly  diffi- 
cult for  students  when  the  material  was  presented  in  the  less  expensive,  programmed 
text  format.  Thus,  a regression  equation  predicting  lesson  completion  time  which 
incorporated  a presentation  medium  variable  would,  under  these  circumstances,  tend 
to  misrepresent  the  utility  of  audio-visual  presentation  as  compared  to 
text  because  data  on  the  audio-visual  mode  were  not  sampled  acros  't  = c* 
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lesson  content.  It  is  anticipated  that  problems  of  this  type  will  continue  to  be  a 
major  limitation  to  further  development  of  the  predictive  model. 
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The  computer  cost  analysis  model  developed  in  this  study  incorporates  an 
estimate  of  students’  course  completion  time  based  on  relevant  course  content, 
instructional  design,  and  student  characteristic  variables.  It  thus  increases  the 
scope  of  Rand's  MODIA  cost  model  by  basing  course  completion  times  on  predicted 
values  from  a regression  equation  derived  from  actual  data  from  an  operational 
environment.  The  flexibility  of  input  preparation  allows  the  model  to  be  used  for 
both  general  and  detailed  levels  of  analysis. 


The  cost  figures  produced  by  the  program  are  only  as  good  as  the  accuracy  of 
the  course  description  provided  and  the  similarity  of  the  course  to  be  modeled  to 
the  AIS  courses  from  which  data  were  sampled  in  this  study.  A further  limitation 
of  the  cost  analysis  program  is  that  only  mean  cost  values  are  produced  even  though 
considerable  variability  is  associated  with  the  predicted  course  completion  times. 
Further  work  on  the  AIDE  cost  model  could  improve  its  usefulness  by  computing  the 
variances  associated  with  the  final  course  cost.  The  inclusion  of  variances  would 
allow  analysis  of  variance  and  linear  model  comparisons  to  be  made  on  the  costs  of 
alternative  approaches.  As  the  model  now  stands,  exercising  it  with  respect  to 
questions  related  to  course  content,  instructional  design,  arid  student  character- 
istics produces  cost  figures  which  can  only  be  compared  in  terms  of  percentage  of 
change  without  any  measure  of  the  significance  of  the  cost  differences. 

In  summary,  the  current  study  has  provided  three  products  which  can  facilitate 
the  investigation  of  cost  savings  due  to  the  individualization  of  military  training. 
The  Descriptive  Model  of  Computer-Based  Training  Systems  will,  hopefully,  provide  a 
context  within  which  to  investigate  the  various  components  of  such  systems.  The 
Predictive  Model  has  explored  a methodology  which,  with  minor  variations,  can  be 
developed  to  predict  timesavings  due  to  individualization  accurately  in  a variety  of 
training  environments.  As  instructional  systems  such  as  the  AIS  become  established, 
a broad,  reliable  data  base  will  become  available  which  can  be  used  to  support  a 
systematic,  empirical  approach  to  the  selection  of  additional  courses  for  individual- 
ization. Finally,  the  AIDE  Cost  Model  has  extended  the  Rand  MODCOM  program  to 
incorporate  the  variable  course  completion  times  characteristic  of  computer-based 
training  systems. 
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One  final  corment  would  appear  to  be  appropriate.  As  was  discussed 
in  Section  5.3  of  this  report.  Discussion  of  AIDE  Example  Run  Results,  those 
training  expenses  which  are  directly  attributable  to  students  and  which  are 
sensitive  to  variations  in  course  completion  time,  (i.e.,  student  pay  and 
allowances)  form  a relatively  small  proportion  of  total  course  costs.  If 
substantive  cost  savings  are  to  be  realized  from  the  individualization  of 
instruction,  the  total  training  system,  including  support  personnel  and 
facilities,  must  be  critically  examined.  For  example,  the  emphasis  to 
date  has  been  on  shortening  course  completion  time  while  holding  the  number 
of  instructional  personnel  constant.  It  may  well  be  more  cost  effective  to 
allow  somewhat  longer  course  completion  times  while  reducing  the  student/ 
instructor  ratio.  The  coordination  of  student  arrivals  and  departures  from 
the  training  base  is  at  least  as  important  as  is  the  efficient  use  of  the 
student's  time  in  the  classroom.  Personnel  whose  duties  are  only  tangentially 
related  to  training  account  for  a sizeable  portion  of  a training  base's 
budget.  Reducing  the  length  of  one  or  two  small  enrollment  courses  by  20 
or  even  30  percent  will  not  appreciably  decrease  the  number  of  students 
on  base  and  will  not,  consequently,  allow  any  reductions  in  the  number 
of  these  support  personnel.  If,  on  the  other  hand,  a substantial  proportion 
of  the  training  on  a given  base  were  individualized  and  the  expected  time 
savings  realized,  a number  of  indirect  costs  could  be  reduced.  Referring 
to  the  functional  cost  items  shown  for  the  standard  sample  in  Table  7,  one 
would  expect  broad  scale  reductions  in  course  times  to  impact  the  following 
items:  Facility  Construction;  Instructor  Pay  and  Allowances;  Facilities 

Maintenance  Personnel;  Training  Administration  Personnel;  Base  Operating 
Support  Personnel;  Medical  Personnel,  PCS  costs  for  instructors;  and  Instructor 
Education  Training.  These  items  account  for  an  additional  21  percent  of  the 
total  course  costs,  and  reductions  in  these  areas  would  almost  certainly 
result  in  substantive  savings.  It  may  well  be  the  case  that  the  full  potential 
of  computer-based  training  systems  for  cost  reduction  will  only  be  realized 
when  they  have  become  the  norm  rather  than  the  exception  in  military  training. 
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GENERAL 

This  appendix  describes  some  technical  aspects  of  the  AIDE  computer  cost 
analysis  program.  Appendix  B gives  a complete  FORTRAN  listing  of  the  AIDE  program. 
Appendix  A includes  the  following: 

0 Computer  Program  Overview 

0 Differences  Between  AIDE  and  MODCOM  Data  Inputs 
0 Using  a different  Regression  Equation 
0 General  Comments  About  the  FORTRAN  Program 
0 Input  Data  Preparation 
0 Computer  Program  Output  Description. 


A-2 


/VrCOOAfWff.1.  DOUGLAS  ASTRONAUTICS  CO  1*1  RANT  - CAST 


EVALUATING 
INSTRUCTIONAL  SYSTEMS 


VOLUME  II 


MDC  El 570 
15  MAY  1977 


Computer  Program  Overview 

The  computer  program  was  called  Aid  for  Instructional  Development  and  Evalua- 
tion (AIDE).  The  development  of  the  AIDE  computer  program  was  facilitated  by  split- 
ting the  program  into  the  following  major  modules: 

Input  Module:  Read  and  check  input  data.  Input  data  will  include  lesson 

descriptors,  student  descriptors,  instructor  data,  courseware  data,  instruc- 
tional resources  data,  physical  facilities  data,  and  CMI  system  data.  A com- 
plete list  of  the  input  data  is  given  in  a subsequent  section  of  this  Appendix. 

Course  Duration  Module:  Use  the  regression  model  and  overhead  factor  developed 

in  Section  4.0  to  compute  the  average  course  duration  and  the  percentage  time- 
saving. 

Student  Flow  Module:  Use  the  average  time-to-complete  the  course  to  compute 

the  average  enrollment  level  in  the  course. 

Resources  and  Personnel  Module:  Based  on  the  average  enrollment  in  the  course 

and  other  user  supplied  course  data,  find  the  courseware  development  and  main- 
tenance manhours,  student  and  instructor  staff  manhours,  instructional  hard- 
ware and  other  resources  procurement  quantities  and  maintenance  manhours,  and 
computer  system  hardware  quantities.  In  determining  these  quantities,  allow- 
ance is  made  for  fluctuations  in  the  enrollment  level. 

Cost  Accumulation  Module:  Combine  personnel  and  resource  quantities  with  unit 

costs  to  arrive  at  development,  initial  investment,  and  annual  operations  and 
maintenance  costs  to  operate  the  course  over  a period  of  five  years. 

Output  Module:  Write  intermediate  and  final  results. 

Prior  to  the  start  of  programming  the  model,  our  literature  survey  and  contacts 
with  instructional  system  developers  revealed  that  the  Rand  Corporation  had  been 
developing  an  instructional  model  called  MODIA  (Methods  of  Designing  Instructional 
Alternatives).  The  MODIA  Cost  Model  (MODCOM)  was  well  developed  and  performed 
several  of  the  functions  needed  in  the  AIDE  Model.  Specifically,  these  are: 

0 Several  input  routines  for  courseware,  hardware  and  personnel 

0 A routine  to  generate  student  manyears,  given  the  average  time  to  complete 
the  course 
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0 Estimation  of  training  administration,  base  operating  support,  and  medical 
support  personnel 

0 A cost  accumulation  subroutine 

0 Routines  to  summarize  input  data,  intermediate  results,  and  final  results. 

In  view  of  these  similarities,  we  requested  and  received  a copy  of  the  MODCOM 
computer  program  from  the  Rand  Corporation. 

The  programming  efforts  were  then  directed  at  writing  the  subroutines  to  compute 
the  course  duration  and  estimate  resource  quantities  and  modifying  the  MODCOM  program 
so  as  to  integrate  it  with  these  subroutines.  A listing  of  the  computer  program  is 
given  in  Appendix  B.  The  inputs  to  be  supplied  by  the  user  and  the  outputs  of  the 
computer  program,  are  described  in  the  following  paragraphs. 

Differences  Between  AIDE  and  MODCOM  Data  Inputs 

The  first  15  data  card  types  of  AIDE  are  similar  to  those  of  MODCOM,  but  AIDE 
adds  a 16th  data  card  type  for  lesson  descriptors.  Specific  format  differences 
exist  in  AIDE  for  the  following  card  types:  3,  9,  10  and  12.  AIDE  always  uses 

the  calculation  option  where  MODCOM  gives  a choice  of  calculation  or  throughput. 

Using  a Different  Regression  Equation 

A regression  equation  of  up  to  three  student  varibables  and  up  to  eleven 
lesson  variables  can  be  entered  by  changing  the  DATA  statement  values  in  lines 
91  through  98  of  the  program  AIDE  with  no  other  reprogramming  necessary.  NSVAR 
and  LDES  are  the  number  of  student  and  lesson  variables  respectively.  TVAL(ll) 
is  the  vector  of  maximum  values  for  the  lesson  variables.  Regression  equations 
of  a different  combination  of  variables  will  require  additional  changes  in  the 
FORTRAN  program  in  PROGRAM  AIDE  and  in  subroutines  (overlays)  READS,  READW,  AIDEA 
and  AIDEB.  Specifically,  the  common  statement  of  AIDEIA  and  A1DE2  will  be  involved 
as  well  as  read  and  write  formats  in  PROGRAM  AIDE  and  subroutines  READS,  READW, 

AIDEA  and  AIDEB. 
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General  Comments  About  the  FORTRAN  Program 

The  AIDE  FORTRAN  Program  was  developed  on  the  CDC  Cyber  73-16  computer  at  AIS, 
Lowry  Air  Force  Base,  Colorado.  The  program  is  presently  overlayed  to  fit  within 
a 130K  core  memory  limit.  The  overlay  will  need  to  be  re-setup  by  a programmer  for 
the  local  computer  facility.  The  flowcharts  and  other  detailed  documentation 
accompanying  the  MODCOM  FORTRAN  program  (Hess  and  Kantar,  1976)  apply  to  AIDE 
as  well,  with  the  exception  that  AIDE  adds  three  subroutines  (READW,  AIDEA,  and 
AIDEB)  and  deletes  one  subroutine  (0UT5). 

Input  Data  Preparation 

This  section  describes  the  inputs  to  the  AIDE  computer  program.  The  general 
form  and  style  of  the  inputs  are  derived  from  the  Rand  MODCOM  program  which  played 
a large  part  in  the  development  of  our  AIDE  model.  The  description  of  the  Rand 
model  variables  is  taken  from  a report  provided  to  us  prior  to  publication  by  Rand. 

A large  amount  of  data  can  be  entered  into  the  program.  Only  data  relating  to 
lessons  (regression  model  variables),  student  variable  scores,  and  class  entry  inter- 
val are  required.  Any  other  data  omitted  will  be  treated  either  as  if  it  were  zero 
or  assigned  a default  value. 

Data  are  input  to  the  model  on  cards.  Sixteen  types  of  card  formats  are  used,  as 
listed  in  Table  11.  The  detailed  form  of  each  card  is  described  below.  The  follow- 
ing general  comments  apply  to  all  cards: 

Format  Identifier:  All  cards  must  have  the  format  type  punched  in  columns 

1 and  2. 

Blanks:  Unless  otherwise  specified,  all  blanks  in  numeric  fields  are  read  as 

zeros. 

Mandatory  Inputs:  The  variables  which  must  be  assigned  non-zero  inputs  are: 
"entry  interval  (in  hours),"  on  card  format  2;  student  variables  scores  on 
card  3;  and  at  least  one  set  of  lesson  descriptors  (card  16).  Failure  to 
enter  a non-zero  value  for  these  variables  will  result  in  the  printing  of  an 
error  message  and  program  termination.  Format  type  99  is  used  to  signify  the 
end  of  the  input  data;  one  card  of  this  type  must  be  present.  Failure  to 
enter  this  card  will  result  in  a FORTRAN  error  message  and  program  termination. 
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Card  Limit:  In  order  to  preserve  core  memory,  it  was  necessary  to  put  limits 

on  the  number  of  cards  of  each  format  type  which  can  be  entered  for  a single 
run  of  the  AIDE  Model.  This  limit  is  indicated  in  the  description  of  each 
card  type.  Cards  in  e;^cess  of  that  limit  are  deleted  from  further  processing. 

Card  Deletions:  Throughout  this  section,  mention  is  made  of  card  deletions 

which  will  occur  if  certain  input  conditions  are  violated  (e.g.,  input  not 
within  acceptable  range,  and  illegal  input  combinations).  All  such  card 
deletions  will  be  listed  as  part  of  the  output  with  an  appropriate  error  message. 

Decimal  Pointy:  All  input  data  are  entered  as  real  (floating-point)  numbers. 

Tne  implied  decimal  point  location  is  at  the  right  end  of  each  field  unless 
otherwise  specified.  However,  a punched  decimal  point  always  overrides  an 
implied  decimal  point  location. 
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Format  1:  Title  Card  Card  Limit  = 1 - This  card  is  used  to  identify  AIDE  runs. 

The  title,  to  be  entered  in  columns  3 through  80,  may  contain  between  1 and  78 
alphanumeric  characters.  If  this  card  is  excluded,  then  the  default  title,  AIDE 
model,  is  used. 

Format  2:  General  Course  Factors  Card  Limit  = 1 - This  card  contains  inputs 

describing  washouts,  instructor  training  costs,  and  the  old  course  du“*ation. 

Columns  3-7  (5  column  numeric  field):  Old  course  duration  (in  hours).  This 

number,  as  described  in  Section  4.2,  is  the  number  of  hours  of  instruction 
presented  in  the  old  course.  If  a value  is  not  entered,  the  new  course 
duration  will  be  computed,  but  not  time  savings. 

Columns  8-11  (4  column  numeric  field):  Washout  rate  (%):  The  ratio  of  the 

number  of  washouts  to  the  total  number  of  entrants  times  100. 

Columns  12-15  (4  column  numeric  field):  Entry  interval  (in  hours):  The 

average  length  of  time  separating  entry  classes. 

Columns  16-19  (4  column  numeric  field):  Number  of  months  instructors  on  board 

in  year  0:  The  number  of  months  the  initial  instructor  cadre  will  spend  pre- 

paring for  the  subject  course  in  year  0.  This  value,  which  must  be  supplied 
by  the  planner,  should  include  time  required  for  special  factory  training  and 
education  training  as  well  as  time  necessary  for  curriculum  development. 

Columns  20-24  (5  column  numeric  field):  Cost  of  initial  factory  training  (in 

dollars):  The  total  cost,  less  student  pay  and  allowances,  of  providing  the 

initial  instructor  cadre  with  specialized  equipment  or  system  training  neces- 
sary for  course  development  and  conduct.  This  initial  specialized  training  is 
usually  accomplished  at  a private  contractor's  facility.  Subsequent  specialized 
training  of  instructors  is  usually  provided  informally  by  the  existing  instruc- 
tor nucleus.  This  cost  should  include  travel  per  diem,  transportation  expenses, 
destination  per  diem,  and  attendance  fees  (or  contract  costs),  as  applicable. 

The  pay  and  allowances  of  the  initial  instructor  cadre  while  in  student  status 
will  be  accounted  for  elsewhere. 

Columns  25-29  (5  column  numeric  field):  Cost  of  instructor  e^^'ication  training 

($):  The  cost  of  the  six-week  instructor  prerequisite  course  on  education 

principles. 
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Format  3:  Student  Inputs  Card  Limit  = 75  - This  card  contains  inputs  relating 

to  the  number  and  type  of  student  entrants,  and  values  of  the  three  student  charac- 
teristics variables  (ASVADM,  ASVMEC,  SEX). 


Breakdown  of  Student  Entrants  - The  course  planner  must  determine  the  breakdown 
of  total  annual  student  entrants  by  personnel  type  (Active  Duty  Force  Pipeline, 

Active  Duty  Force  Lateral/Upgrade,  Guard/Reserve  Pipeline,  Guard/Reserve  Lateral/ 
Upgrade,  Other  DoD  Pipeline,  Other  DoD  Lateral/Upgrade,  Non-DoD),  personnel  designator 
(officer,  airman,  civilian),  and  paygrade.  Typical  paygrades  for  Air  Force  pipeline 
and  lateral /upgrade  students  are: 

Student  Type  Officer  Ai rman 

Pipeline  0-1  E - 1 

Lateral  or  Upgrade  0-2  thru  0-4  E - 3 thru  E - 5 

Columns  3-4  (2  column  numeric  field):  Personnel  Type:  Numeric  personnel  type 

code  as  follows: 

01  = Active  Duty  Force  Pipeline  Student 

02  = Active  Duty  Force  Lateral  Upgrade 

03  = Guard/Reserve  Pipeline  Student 

04  = Guard/Reserve  Lateral  Upgrade  Student 

05  = Other  DoD  (Army,  Navy)  Pipeline  Student 

06  = Other  DoD  (Army,  Navy)  Lateral  Upgrade  Student 

07  = Non-DoD  (HAP)  Student 

Use  of  a code  outside  the  range  1 through  7 will  result  in  the  deletion  of  the 
card. 


Columns  5-6  (2  column  numeric  field):  Personnel  Designator;  Numeric  personnel 

designator  code  as  follows;  ' • 

1 = Officer 

2 = Ai rman 

3 = Civilian-GS  (General  Schedule) 

4 = Civil ian-WB  (Wage  Board) 

5 = Non-DoD 

Use  of  a code  outside  the  range  1 through  5 will  result  in  the  deletion  of  that 
card.  Additionally,  card  deletion  will  occur  if  personnel  designator  5 is  used 
in  combination  with  any  personnel  type  other  than  7. 

A- 8 

n/fCOOMMCLL.  DOUGLAS  ASTftO/MAUTICS  COfVIPAf>iV  - CAS  T 


EVALUATING  MDC  E1570 

INSTRUCTIONAL  SYSTEMS  VOLUME  II  15  MAY  1977 

Columns  7-8  (2  column  numeric  field):  Pay  Grade:  Standard  U.S.  Government 
numeric  pay  grade  codes.  The  code  entered  in  these  columns  must  be  within  the 
range  of  permissible  paygrades  for  the  specified  personnel  designator  or  else 
the  card  will  be  deleted.  A duplicate  personnel  type/personnel  designator/ 
paygrade  combination  will  also  cause  a card  to  be  deleted.  The  number  of 
unique  paygrades  entered  for  a given  personnel  type/personnel  designator  com- 
bination will  depend  on  the  required  level  of  accuracy  and  the  ambition  of  the 
user:  each  paygrade  requires  a separate  card  entry.  ATC  generally  uses  a 
single  paygrade  (the  modal)  for  each  personnel  type/personnel  designator  com- 
bination. In  order  to  reduce  the  planner's  input  requirements,  pay  and 
allowance  factors  for  each  paygrade  have  been  stored  as  part  of  the  model. 

Civilian  (Personnel  Designator  = 3 and  4) 

Stored  P&A  Factor  ($)  Stored  P&A  Factor  ($) 

Paygrade  for  GS  Ci vi 1 ians  (3)  for  WB  Ci vi 1 ians  (4) 


18 

39,060 

17 

39,060 

16 

39,060 

15 

35,823 

14 

30,377 

15,157 

13 

28,053 

14,773 

12 

22,285 

13,845 

n 

18,935 

13„373 

10 

17,041 

12,758 

9 

15,717 

12,127 

8 

14,254 

11 ,471 

7 

13,828 

11,077 

6 

11,782 

10,527 

5 

10,483 

9,587 

4 

9,266 

9,214 

3 

7,953 

8,896 

2 

6,447 

8,491 

1 

5,651 

7,912 

Blank  or  0 

12,599 

11,464 

(Overal 1 

Average) 

Additionally,  a dummy  pay  and  allowance  schedule  has  been  set  up  for  non-DoD 
students  with  20  possible  paygrades.  However,  because  of  the  wide  range  of  possible 
pay  rates,  typical  values  could  not  be  established;  and  consequently,  the  stored 
values  were  set  equal  to  zero.  If  the  planner  feels  that  non-DoD  student  pay  and 
allowances  are  relevant  to  a particular  costing  exercise,  then  the  pay  factor  over- 
ride option  should  be  employed  to  introduce  the  appropriate  pay  rate. 
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Columns  9-11  (3  column  numeric  field):  The  score  of  this  type  of  student  on 

the  ASVADM  test  (0-100). 

Columns  12-14  (3  column  numberic  field):  The  score  of  this  type  of  student  on 

the  ASVMEC  test  (0-100). 

Columns  15-17  (3  column  numeric  field):  The  ratio  of  male  students  to  the 

sum  of  male  and  female  students  (0-1.0). 

Columns  18-37  (five  4-column  fields):  Number  of  student  entrants  of  this  type 

in  years  1-5. 

Format  4:  Instructor  Inputs  Card  Limit  = 25  - This  card  contains  inputs 

pertaining  to  the  size,  turnover,  and  student  loading  of  the  instructor  force. 

Breakdown  of  Instructor  Force  - The  course  planner  must  determine  the  breakdown 
of  total  instructor  force  by  personnel  type  (Air  Force,  Other  DoD),  personnel  desig- 
nator (officer,  airman,  civilian)  and  paygrade,  and  student  load.  Typical  paygrades 
for  each  instructor  type  are  provided  in  Table  12. 

TABLE  12 


TYPICAL  INSTRUCTOR  PAYGRADES 


Instructor  Type 

Officer 

Ai rman 

Civilian 

Academic,  remedial  or  special 
requirements  instructor 

Evaluators  and  Supervisors 

0 - 3 
0 - 4 

3 - 5 

E - 6.  E - 7 

GS  - 9 
GS  - 11 

Columns  3-4  (2  column  numeric  field):  Personnel  Type:  Numeric  personnel  type 

code  as  follows: 

8 = Air  Force  Instructor 

9 = Other  DoD  (Army,  Navy)  Instructor 

Use  of  a code  other  than  8 or  9 will  result  in  the  deletion  of  that  card. 
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Columns  5-6  (2  column  numeric  field):  Personnel  Designator:  Numeric  oersonnel 

designator  code  as  follows: 


1 = Officer 

2 = Airman 

3 = Civilian  - GS 

4 = Civilian  - WB 

Use  of  a code  outside  the  range  1 through  4 will  result  in  the  deletion  of 
that  card. 


Columns  7-8  (2  column  numeric  field):  Paygrade:  Standard  U.S.  Government 

numeric  paygrade  codes.  The  codes  and  stored  pay  factors  are  the  same  as 
those  described  for  input  format  3.  As  before,  the  code  entered  in  these 
columns  must  be  within  the  range  of  permissible  paygrades  for  the  specific 
personnel  designator  or  else  the  card  will  be  deleted.  A duplicate  personnel 
type/personnel  designator/paygrade  combination  will  also  cause  a card  to  be 
deleted. 

Columns  9-11  (3  column  numeric  field):  Turnover  rate  (%):  The  percentage  of 

instructors  who  will  be  replaced  each  year  or  the  reciprocal  of  the  average 
instructor's  tour  length  in  years  times  100. 

Columns  12-16  (5  column  numeric  field):  Size  of  initial  instructor  cadre  in 

year  0;  the  number  of  instructors  (not  manyears)  brought  onboard  in  year  0 for 
curriculum  development  and  any  required  fartory  training. 

Columns  17-19  (3  column  numeric  field):  Students  per  Instructor:  The  number 

of  students  that  can  be  instructed  or  counseled  by  an  instructor  of  thi*^  type. 


Format  5:  Courseware  Procurement  Inputs  Card  Limit  = 75  - The  model  recognizes 
four  classes  of  courseware:  (1)  printed  media  such  as  textbooks  and  workbooks  which 
are  intended  for  both  students  and  instructors;  (2)  printed  media  such  as  lesson  and 
evaluation  guides  which  are  intended  only  for  instructors;  (3)  display  media  such  as 
films,  slides,  and  charts;  and  (4)  software  such  as  teaching  machine  programs. 


Four  separate  costs  are  estimated:  (1)  The  initial  cost  of  producing  the  master 

from  which  all  subsequent  copies  will  be  made;  (2)  the  cost  of  reproducing  and  pack- 
aging the  required  number  of  copies;  (3)  the  annual  cost  of  revision  due  to  change 
in  course  content;  and  (4)  the  annual  cost  of  replacement  due  to  loss,  damage,  or 
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normal  aging.  The  cost  of  courseware  development,  such  as  script  writing  and  edit- 
ing, is  accounted  for  within  curriculum  manpower  pay  and  allowances. 

Columns  3-4  (2  column  numeric  field):  Courseware  ID  Number:  Unique  number  for 

each  courseware  type  in  range  1 through  99.  A card  will  be  deleted  if  its  ID 
number  duplicates  that  of  another  card  or  if  its  ID  number  falls  outside  the 
prescribed  range. 

Column  5 (1  column  numeric  field):  Class:  Numeric  courseware  class  code  as 

fol lows : 

1 = Printed  media  for  students  and  instructors  (texts,  workbook) 

2 = Printed  media  for  instructors  only  (lesson  and  evaluation  guide) 

3 = Display  media  (films,  slides) 

4 = Software  (teaching  machine  programs) 

Use  of  a code  outside  the  range  1 through  4 will  result  in  the  deletion  of  that 
card. 

Columns  6-27  (22  column  alphanumeric  field):  Name  of  courseware:  User- 

selected  name  for  identifying  each  courseware  type. 

Columns  28-31  (4  column  alphanumeric  field):  Name  of  courseware  copies: 

User-selected  name  for  identifying  the  unit  denominator  (book,  reel,  set)  of 
each  courseware  type. 

Columns  32-37  (6  column  alphanumeric  field):  Name  of  courseware  measure: 

User-selected  name  for  identifying  the  unit  of  measure  (pages,  minutes,  slides, 
etc.)  for  each  courseware  type. 

Columns  38-41  (4  column  numeric  field);  Total  number  of  courseware  measure  per 
copy;  The  number  of  pages,  slides,  or  minutes,  etc.,  per  unit  of  courseware. 

Columns  42-46  (5  column  numeric  field):  Initial  preparation  cost  per  course- 
ware measure  ($):  The  cost  of  producing  a single  measure  of  master  courseware. 

Some  typical  values  are  provided  in  Table  13. 

Columns  47-50  (4  column  numeric  field);  Copy  cost  per  courseware  measure  ($); 
The  cost  of  copying  a single  measure  of  master  courseware.  Typical  values  are 
again  provided  in  Table  13. 
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Columns  51-54  (4  column  numeric  field):  Packaging  cost  per  unit:  The  cost  :f 

cassettes,  reels,  trays,  etc.,  per  unit  of  courseware.  Some  typical  values 
are  as  follows: 


Sound  on  Slide'  Tray 

$30.00 

Slide  Tray  - 40  slides,  straight 

2.50 

Audiotape  Cassette 

2.00 

Video  Cassette  Tape  (30  minutes) 

18.00 

Video  Cassette  Tape  (60  minutes) 

25.00 

Loose-leaf  Binder  (3-12", 
Telescoping  Poles) 

3.00 

TABLE  13 

TYPICAL  COURSEWARE  PROCUREMENT  COSTS 


Courseware 

Measure 

Ini tial 

Preparation  Cost/ 
Courseware  Measure 
Dol  lars 

Copy  Cost/ 
Courseware  Measure 
Do  1 1 a rs 

Printed  Text 

Pages 

$1.50  - $4.50 

.01 

Sound  Slide  Set  (Color) 

SI i des 

50  - 1.00 

.08 

35mm  Color  Slide  (Text) 

Slides 

4.50  - 9.00 

.08 

35mm  Color  Slide  (Illustration) 

SI  ides 

7.50  - 25.00 

.08 

8mm  Animated  Sequence 

Seconds 

.40  - .85 

- 

Video  Tape 

Mi nutes 

1.00  - 2.00 

- 

Columns  55-57  (3  column  numeric  field):  Annual  revision  rate  (%):  The  per- 
centage of  each  courseware  type's  actual  presentation  minutes  which  can  be 
expected  to  be  revised  annually  due  to  changes  in  course  content.  These 
changes  in  course  content  could  be  caused  by  such  things  as  modifications  to 
mission  equipment,  a required  upgrading  of  student  capabilities,  or  an  effort 
to  improve  the  utilization  of  course  resources.  The  model  assumes  that  after 
the  necessary  revisions  are  made  to  the  master,  sufficient  copies  will  be  made 
of  the  revised  portions  (only)  and  integrated  into  the  existing  copies. 

Columns  58-60  (3  column  numeric  field):  Annual  replacement  rate  (%}:  The 

percentage  of  each  courseware  type's  units  which  will  need  to  be  replaced 
annually  due  to  damage  or  normal  aging. 
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Columns  61-64  (4  column  numeric  field):  The  number  of  lessons  in  the  course 

utilizing  this  courseware.  Courseware  description  need  not  limit  it  to  only 
one  lesson.  For  instance,  several  programmed  text  lessons  could  be  grouped 
into  one  data  entry.  If  blank,  value  is  set  equal  to  one. 

Columns  65-67  (3  column  numeric  field):  Number  of  simultaneous  users  per  copy. 
This  entry  is  used  for  courseware  that  is  used  simultaneously  by  more  than  one 
student,  as  in  a four-man  AV  presentation.  If  blank,  value  is  set  equal  to  one. 


Format  6:  Curriculum  Manpower  Inputs  Card  Limit  = 75  - Curriculum  personnel 

determine  training  requirements  and  develop  course  written  material.  Normally, 
attached  to  the  Curriculum  Unit  of  a training  department,  these  personnel  share  the 
task  of  courseware  development  with  the  course's  instructors. 

Columns  3-4  (2  column  numeric  field):  Courseware  ID  number:  The  number  used 

to  identify  the  same  courseware  type  in  input  format  5.  A format  6 card  will 
be  deleted  if  its  ID  number  does  not  have  a match  on  a format  5 card,  or  if 
its  ID  number  duplicates  that  of  another  format  6 card. 

Columns  5-9  (5  column  numeric  field):  Number  of  course  hours:  The  number  of 
course  hours  taken  up  by  a given  courseware  type.  This  number  should  be 
coordinated  with  the  number  of  lessons  specified  on  card  type  5 for  this  type 
of  material . 

Columns  10-14  (5  column  numeric  field):  Number  of  curriculum  manhours  per 

course  hour:  The  number  of  initial  development  curriculum  manhours  required 

for  each  course  hour.  Generally  speaking,  this  number  is  a function  of  the 
teaching  agent  and  the  teaching  format  and  is  independent  of  the  courseware  type. 
Typical  values  are  provided  in  Table  14. 


Columns  15-47  (3  column  numeric  field):  Percentage  accomplished  by  instructors:  , 

The  percentage  of  curriculum  development  done  by  the  instructors  rather  than 
curriculum  personnel. 


Format  7:  Hardware  Procurement  Inputs  Card  Limit  = 75  - The  model  estimates 

four  types  of  hardware  costs:  (1)  the  cost  of  purchasing  the  inc*-emental  number  of 

units  required  each  year,  including  initial  replacement  part  stocks;  (2)  credits  for 
any  surplus  items  returned  to  the  Stock  Fund;  (3)  the  annual  cost  of  replacing  those 
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TABLE  14 

INITIAL  COURSEWARE  DEVELOPMENT 
MANHOURS  PER  CONVENTIONAL  COURSE  HOUR 


Presentation 

Development* 

Media 

Manhours  per  Hour 

Programmed  Text 

50  - 70 

Audio  Visual 

80  - 1 00 

Computer  Assisted  Instruction 

100  - 300 

*If  new  material  is  being  written,  without  benefit  of 
notes  from  the  old  course,  these  numbers  should  be 
increased  by  about  40%. 


units  worn  out  during  the  year;  and  (4)  the  annual  cost  of  misecl laneous  repair 
parts. 

Columns  3-4  (2  column  numeric  field):  Hardware  ID  number:  Unique  number  for 

each  hardware  type  in  range  1 through  99.  A card  will  be  deleted  if  its  ID 
number  duplicates  that  of  another  card  if  its  ID  number  falls  outside  the 
prescribed  range. 

Column  5 (1  column  numeric  field):  Class:  Numeric  hardware  class  code  as 

follows: 

1 = media  hardware  (slide  projectors,  reel-to-reel  projectors,  tape 

recorders,  teaching  machines) 

2 = special  equipment  (trainers,  bailed  aircraft) 

3 = overhead  hardware  (chairs,  desks) 

Use  of  a code  outside  the  range  1 through  3 will  result  in  the  deletion  of  that 
card. 

Columns  6-35  (30  column  alphanumeric  field):  Name  of  hardware  type:  User- 

selected  name  for  identifying  each  hardware  type. 

Columns  36-39  (4  column  numeric  field):  The  number  of  units  of  a given  hard- 

ware type  which  are  available  to  a course  at  its  inception,  at  zero  cost. 

Columns  40-47  (8  column  numeric  field):  Procurement  cost  per  unit  ($). 

Normally,  an  AF  planner  will  consult  his  own  service's  supply  manual  to 
determine  hardware  unit  costs. 
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Col umn  48  (1  column  numeric  field):  Credit  Option:  Option  to  obtain  credit 

for  surplus  equipment  returned  to  Stock  Fund  (yes  = 1 ; no  = 0 or  blank).  The 
planner  may  obtain  credit  for  surplus  equipment  returned  to  the  Stock  Fund 
provided  both  the  following  conditions  are  met: 

a.  The  unit  cost  does  not  exceed  $1,000. 

b.  A demand  exists  elsewhere  in  the  AF  for  the  returned  items. 

The  option  should  not  be  exercised  if  either  the  unit  cost  exceeds  $1,000  or 
if  other  AF  demand  does  not  exist.  In  fact,  if  the  planner  enters  a "1"  for 
an  item  costing  more  than  $1,000,  then  the  model  automatically  resets  the 
option  designator  value  to  "0."  The  planner  should  also  enter  a zero  if  any 
surpluses  generated  in  a given  year  are  to  be  retained  for  increased  demands 
in  subsequent  years.  The  amount  of  the  credit  is  60%  of  the  original  purchase 
price,  the  other  40%  assumed  to  go  into  rehabilitation  action  so  that  the 
equipment  may  again  be  sold  at  the  full  purchase  price. 

Columns  49-51  (3  column  numeric  field):  Annual  attrition  rate  (%):  The  per- 

centage of  hardware  units  which  will  need  to  be  replaced  each  year.  This  value 
can  be  approximated  by  multiplying  the  reciprocal  of  the  estimated  average  life 
(in  years)  by  100. 

Columns  52-57  (6  column  numeric  field):  Miscellaneous  repair  part  cost  per 

unit  per  year  ($):  The  annual  cost  of  purchasing  repair  parts  for  a single 

unit.  In  the  absence  of  better  data,  a value  in  the  range  of  6 to  10%  of  the 
unit  procurement  cost  should  be  used. 

Columns  58-61  (4  column  numeric  field):  Number  of  lessons  using  this  type  of 

hardware.  More  than  one  lesson  may  utilize  the  same  type  of  hardware  (such  as 
an  AV  projector);  this  item  allows  the  planner  to  specify  this  fact. 

Columns  62-64  (3  column  numeric  field):  The  number  of  students  who  may  simul- 

taneously use  hardware  item.  This  is  the  number  of  students  who  are  studying 
the  same  lesson  that  may  simultaneously  use  this  hardware  device. 


Format  8:  Hardware  Maintenance  Manpower  Inputs  Card  Limit  = 75  - Hardware 

maintenance  personnel  are  responsible  for  the  preventive  and  corrective  maintenance 
of  course  hardware.  Media  hardware  maintenance  is  generally  the  responsibility  of 
the  Training  Services  Division  (school-level),  while  all  other  types  of  hardware  are 
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usually  serviced  by  the  base  Maintenance  and  Supply  Group  or  a specialized  mainte- 
nance squadron. 

Columns  3-4  (2  column  numeric  field):  Hardware  ID  number:  The  number  used  to 

identify  the  same  hardware  type  in  input  format  7.  A format  8 card  will  be 
deleted  if  its  ID  number  does  not  have  a match  on  a format  7 card  or  if  its 
ID  number  duplicates  that  of  another  format  8 card. 

Columns  5-10  (6  column  numeric  field):  Number  of  failures  per  hour  of  use: 

The  average  number  of  times  a particular  type  of  equipment  can  be  expected  to 
fail  for  each  hour  of  use. 

Columns  11-15  (5  column  numeric  field):  Average  time  to  repair  per  failure 

(in  hours).  This  value  should  include  only  that  portion  of  downtime  in  which 
maintenance  personnel  are  actively  pursuing  the  repair  activity. 

Format  9:  Facility  Procurement  Inputs  Card  Limit  = 25  - This  card  contains 

inputs  relating  to  the  modification  and  construction  of  course  facilities. 

Columns  3-4  (2  column  numeric  field):  Facility  ID  number:  Unique  number  for 

each  facility  type  in  range  1 through  99.  A card  will  be  deleted  if  its  ID 
number  duplicates  that  of  another  card  or  falls  outside  the  prescribed  range. 

Columns  5-34  (30  column  alphanumeric  field):  Name  of  facility  type:  User- 

selected  name  for  identifying  each  facility  type. 

Columns  35-38  (4  column  numeric  field):  Units  available  "as  is"  at  start  of 

first  year:  The  number  of  units  of  a given  facility  type  which  are  available 

to  a course  at  its  inception  at  zero  cost. 

Columns  39-45  (7  column  numeric  field):  Construction  cost  per  unit  ($):  The 

cost  of  constructing  or  modifying  a single  facility  unit. 

NOTE:  If  unit  requirements  for  a given  facility  type  are  assumed  to  be  avail- 

able at  zero  cost  for  all  years  under  study,  then  the  user  should  enter 
a unit  construction  cost  of  zero. 

Columns  46-49  (4  column  numeric  field):  The  number  of  students  who  can  simul- 

taneously use  the  facility.  An  example  is  the  number  of  students  who  can 
simultaneously  use  a laboratory. 
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Format  10:  Facility  Manpower  Inputs  Card  Limit  = 25  - Facility  maintenance 

personnel  are  responsible  for  the  maintenance  and  repair  (including  custodial  ' 

service)  of  course  facilities. 

Columns  3-4  (2  column  numeric  field):  Facility  ID  number:  The  number  used  to  i 

identify  the  same  facility  type  in  input  format  9.  A format  10  card  will  be 

deleted  if  its  ID  number  does  not  have  a match  on  a format  9 card  or  if  its  ID  : 

I 

number  duplicates  that  of  another  format  10  card.  * 

Columns  5-10  (6  column  numeric  field):  Square  feet  per  facility  unit.  j 

. f 

Columns  11-16  (6  column  numeric  field):  Maintenance  manhours  per  square  foot 

per  month.  } 


Format  11:  Training  Administrative,  Base  Operating  Support,  and  Medical 

Manpower  Inputs  Card  Limit  = 3 - This  card  contains  inputs  required  for  the  deri- 
vation of  training  administrative,  base  operating  support,  and  medical  manpower. 

Training  administrative  personnel  perform  training-related  overhead  functions 
not  otherwise  accounted  for.  This  includes  functions  at  the  school-level 
(Administrative  Affairs  Office),  at  the  department  level  (Administrative  Section, 
Requirements  Unit,  and  Instruction  and  Measurement  Unit),  and  at  the  branch  level 
(branch  administration).  It  does  not  include  the  functions  provided  by  the  Training 
Services  Division  or  the  Instructor  Training  Division  (school  level)  or  the 
Curriculum  Unit  (department  level),  however. 

Base  operating  support  personnel  perform  the  following  base  housekeeping  and 
service  functions:  supply,  transportation,  security  police,  material  staff,  comp- 

troller, personnel  staff,  food,  commissary,  housing,  laundry  and  dry  cleaning, 
recreation,  education,  transient  aircraft  maintenance,  and  general  (not  course 
specific)  facility  maintenance.  Medical  personnel  provide  medical  and  dental  care 
to  military  personnel. 

Columns  3-4  (2  column  numeric  field):  Personnel  Type:  Numeric  personnel  type 

code  as  follows: 

13  = Training  Administrative 

14  = Base  Operating  Support 

15  = Medical 
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Columns  5-6  (2  column  numeric  field):  Set  to  01. 

Columns  7-11  (5  column  numeric  field):  Number  of  fixed  manyears  per  course  for 

support  services. 

Columns  12-16  (5  column  numeric  field):  Number  of  variable  manyears  per  Type  A 

manyear  for  support  services. 

Columns  17-21  (5  column  numeric  field):  Number  of  variable  manyears  per  Type  B 

manyear  for  support  services. 

Typical  values  for  these  parameters  are  shown  below.  Type  A manyears  are  the 
sum  of  student,  instructor,  and  curriculum  personnel  manyears.  Type  B manyears 
are  the  military  TOY  student  manyears. 

FIXED  MANYEARS  VARIABLE  MANYEARS  VARIABLE  MANYEARS 

PER  COURSE PER  TYPE  A MANYEAR  PER  TYPE  B MANYEAR 

Training  Administrative  3 .05 

Base  Operating  Support  0 .08  .035 

Medical  0 .02  .005 

Format  12:  Computer  Management  System  Inputs  Card  Limit  = 3 - The  three  hard- 

ware items  of  the  computer  management  system  are: 

CLASS  ITEM 

01  Mainframe  (CPU  and  Main  Memory) 

02  Management  Terminals 

03  Auxiliary  Storage 

For  each  of  these,  the  following  data  are  entered: 

0 Hardware  class:  as  designated  above 

0 Hardware  name:  user  selected  name 

0 Capacity  of  unit:  for  mainframe  and  management  terminals,  this  is  the 

number  of  CMI  transactions  per  hour;  for  the  auxiliary  storage  it  is  the 
number  of  student  data  sets  (complete  records  of  one  student)  that  can  be 
stored. 

0 Yearly  cost  of  unit:  Total  yearly  cost  of  the  unit;  if  leased  or  purchased, 

maintenance  costs  should  be  included. 

Columns  3-5  (one  column  field):  Hardware  class. 

Columns  6-35  (30  column  alphanumeric  field):  Name  of  hardware  device. 
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Format  13:  Optional  Officer/Airman/Civilian  Distribution  Overrides  - This  card 

provides  the  planner  the  option  of  overriding  stored  officer/airman/civilian  distri- 
bution percentages.  It  is  one  of  two  input  formats  on  which  blanks  and  zeroes  are 
differentiated.  Thus,  the  planner  needs  to  make  entries  only  in  the  fields  of  those 
personnel  types  whose  distribution  he  wishes  to  override;  the  other  fields  should  be 
left  blank.  If  the  override  values  for  a given  personnel  type  do  not  total  100, 


then  the  model  reinstates  the  stored  values.  The  stored  values  are  as  follows: 


Officer 

% 

Ai rman 

% 

Civilian 

% 

Curriculum 

0 

63 

37 

Hardware  Maintenance 

2 

12 

26 

Facility  Maintenance 

0 

100 

0 

Training  Administrative 

6 

39 

55 

Base  Operating  Support 

2 

64 

34 

Medical 

20 

80 

0 

I 

i 


Columns  3-11  (three  3-column  numeric  fields):  Curriculum  manpower  officer/ 

airman/civilian  distribution. 

Columns  12-20  (three  3-column  numeric  fields):  Hardware  maintenance  manpower 

officer/airman/civil ian  distribution. 

Columns  21-29  (three  3-column  numeric  fields):  Facility  maintenance  manpower 

offi cer/airman/ci vi 1 ian  distri but ion. 


Columns  30-38  (three  3-column  numeric  fields): 
power  officer/airman/civilian  distribution. 

Columns  39-47  (three  3-column  numeric  fields): 
off icer/ai rman/ci vi 1 ian  distribution. 

Columns  48-56  (three  3-column  numeric  fields): 
civilian  distribution. 
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Format  14: Optional  Miscellaneous  Overrides  - This  card  allows  the  planner  to 

override  miscellaneous  time  and  cost  factors  which  have  been  stored  as  part  of  the 
program.  It  is  the  second  of  two  input  formats  on  which  blanks  and  zeroes  are 
differentiated.  Thus,  as  before,  the  planner  needs  to  make  entries  only  in  the 
fields  of  those  factors  which  he  wishes  to  override;  the  other  fields  should  be  left 


blank.  The  stored  values  are  as  follows: 

Available  Productive  Manhours/Month 

Hardware  Maintenance  Personnel  121 

All  other  Personnel  Types  142 

Average  Number  of  Training  Days  per  Month  21.7 

Classroom  Training  Hours  per  Student  per  Day  7 

Miscellaneous  Supply  Cost  per  Manyear  ($)  112 

PCS  Cost  per  Move  ($) 

Officer  879 

Airman  224 

Civilian  879 

TDY  Expense  ($) 

One-Way  Transportation  85 

Destination  per  Diem 

Officer  11 

Airman  4 

Civilian  21 


Columns  3-10  (two  3-column  numeric  fields):  Available  productive  manhours  per 

month.  Most  AF  training  personnel  have  142  hours  per  month  available  to  per- 
form productive  work  (174  assigned  hours  less  32  hours  for  leave,  medical, 
training,  and  organizational  duties).  Hardware  maintenance  personnel,  however, 
must  spend  part  of  that  142  hours  waiting  for  parts  and  tools.  This  waiting 
time  is  estimated  at  15«  of  available  hours,  thus  reducing  hardware  maintenance 
available  productive  manhours  per  month  to  121. 

Columns  11-14  (4  column  numeric  field):  Average  number  of  training  days  per 

month. 

Columns  15-18  (4  column  numeric  field):  Classroom  training  hours  per  student 

per  day.  This  value  should  coincide  with  the  value  listed  on  Worksheet  1. 
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Columns  19-23  (5  column  numeric  field);  Miscellaneous  supply  cost  per  student 
per  manyear:  The  cost  of  personnel  supplies  (food,  clothing,  etc.)  and  general 

office  supplies  (paper,  pencils,  forms,  etc.)  per  manyear  (students  and 
permanent  party). 

Columns  24-35  (three  4-column  numeric  fields):  PCS  (Permanent  Change  of 

Station)  cost  per  move  ($):  Includes  transportation  of  personnel  and  depen- 

dents; shipment  and/or  storage  of  household  goods;  and  mileage  per  diem  and 
subsistence  allowance  while  in  travel  status. 

Columns  36-39  (4  column  numeric  field);  Average  TOY  one-way  transportation 
cost  ($):  Average  cost  of  transportation  and  travel  per  diem  to  or  from 

technical  training  center. 

Columns  40-48  (three  3-column  numeric  fields):  TOY  destination  per  diem  ($): 

Average  student  cash  entitlement  while  at  technical  training  centers,  after 
deductions  for  quarters  and  messing  (when  available). 

Format  15:  Optional  Pay  and  Allowance  Overrides  - This  card  permits  the  planner 

to  override  the  stored  pay  and  allowance  factors.  The  stored  values  are  listed  in 
Figure  10. 

Columns  3-4  (2  column  numeric  fields):  Personnel  designator:  Numeric  personnel 

designator  code  as  follows: 

1 = Officer 

2 = Airman 

3 = Civilian  - GS 

4 = Civilian  - WB 

5 = Non-DoD 

Use  of  a code  outside  the  range  1 through  5 will  result  in  the  deletion  of  that 
card. 

Columns  5-6  (2  column  numeric  field):  Paygrade:  Standard  U.S.  Government 
numeric  paygrade  codes.  Card  deletion  will  occur  if  the  paygrade  entered  is 
not  within  the  range  of  permissible  paygrades  for  that  personnel  designator  or 
if  the  personnel  designator/paygrade  combination  duplicates  that  of  another 
card. 
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Personnel  Designator 
Officer 
Ai rman 

Civilian  - GS 
Civilian  - WB 
Non-DoD 

Columns  7-11  (5  column  numeric  field); 
Annual  pay  and  allowances  per  manyear. 
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Permissible  Paygrade  Range 
0 or  blank,  1-10 
0 or  blank,  1-9 
0 or  blank,  1-18 
0 or  blank,  1-14 
0 or  blank,  1-20 

Annual  pay  and  allowance  factor  ($): 


Format  16:  Lesson  Descriptor  Cards  Card  Limit  = 75  - This  card  is  used  to 

describe  each  lesson  in  the  new  self-paced  course.  Thirteen  descriptors  are  needed 
for  each  lesson.  These  variables  are  described  in  paragraph  4.2.2  of  the  report. 

Columns  3-5  (3  column  numeric  field):  Lesson  ID  number;  a sequencing  number 

assigned  by  the  planner.  This  is  used  only  for  convenience  in  arranging  the 
lessons.  It  is  not  used  in  the  computations. 

Columns  6-25  (20  column  alphanumeric  field):  Lesson  name;  user  selected  name 

for  lesson. 

Columns  26-30  (5  column  numeric  field):  The  number  of  lecture/discussion 

minutes  in  the  conventional  course  (CLDM).  Taken  directly  from  the  POI. 

Columns  3,-35  (5  column  numeric  field);  The  number  of  demonstration/perform- 
ance-cognitive minutes  in  the  conventional  course  lesson  (CDPMC).  Taken  directly 
from  the  POI. 

Columns  36-40  (5  column  numeric  field):  The  nunter  of  minutes  of  memory 

content  in  the  lesson  (MEM).  Provided  by  the  course  developer. 

Columns  41-45  (5  column  numeric  field):  The  numerical  rating  from  0 to  5 of 

the  simplicity  or  complexity  of  the  memory  content  in  the  lesson  (MSC). 

Provided  by  the  course  developer. 

Columns  46-50  (5  column  numeric  field):  The  number  of  self-check  items  in  the 

self-paced  course  lesson  (SCKC).  A design  variable. 

Columns  51-55  (5  column  numeric  field):  The  number  of  minutes  of  psychomotor 

task  content  in  the  lesson  (PM).  Provided  by  the  course  developer. 


1 
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Columns  56-60  (5  column  numeric  field): 
audiovisual  minutes  in  the  conventional 
from  the  POI. 

Columns  61-65  (5  column  numeric  field): 
psychomotor  minutes  in  the  conventional 
from  the  POI. 


The  number  of  programmed  instruction/ 
course  lesson  (CPIAVH).  Taken  directly 

The  number  of  demonstration/performance- 
course  lesson  (CDPMP).  Taken  directly 


Columns  66-70  (5  column  numeric  field):  The  numerical  rating  from  0 to  5 of 

the  simplicity  or  complexity  of  the  psychomotor  content  in  the  lesson  (PMSC). 
Provided  by  the  course  developer. 

Columns  71-75  (5  column  numeric  field):  The  number  of  minutes  of  cognitive 

content  in  the  lesson  (COG).  Provided  by  the  course  developer. 

Columns  76-80  (5  column  numeric  field):  The  number  of  outside  assignment 

minutes  in  the  conventional  course  (COUTM).  Taken  directly  from  the  POI. 


Format  99:  Termination  Card  - No  entries  other  than  the  "99"  in  columns  1 and 

2 need  be  made. 


Computer  Program  Output  Description 

The  output  from  the  computer  program  is  illustrated  in  this  section  through  the 
use  of  an  example  run  from  the  program.  Since  much  of  the  output  format  remains  as 
it  was  designed  by  Rand  for  the  MODCOM  Program,  we  have  summarized  their  descriptions 
as  in  MODIA:  The  Cost  Model  (Hess  & Kantar,  1976).  The  user  of  the  model  will  note 

that  some  small  rounding  errors  occur  in  the  totals. 

Output  1 (Figure  A-1)  - This  is  an  exact  echo  of  the  input  data.  Card  columns 
are  shown  on  the  top  of  the  page  so  any  misalignment  of  data  or  input  is  apparent. 

Output  la  (no  figure)  - This  section  contains  error  messages  relating  to  data 
that  were  found  to  be  outside  the  specified  range,  or  from  duplicate  cards.  Fatal 
errors,  such  as  no  lesson  descriptors  or  the  omission  of  the  end  of  data  card 
(Format  99)  will  also  appear  here. 

Output  2 (Figure  A-2  - This  output  lists  the  values  assigned  to  each  variable 
in  the  lesson  descriptor  set.  Only  those  lessons  which  have  passed  the  tests  for 
acceptable  range  appear  here. 
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Output  3 (Figure  A-3)  - This  table  lists  the  computed  time  to  complete  each 
lesson  for  students  with  various  student  variable  values.  Times  are  shown  i"  hours. 
Total  course  time  for  each  time  student  is  shown,  as  well  as  the  weighted  average 
time  to  compelte  the  course,  and  the  percentage  timesaving  (if  TPOI  is  given). 

Output  4 (Figure  A-4)  - This  output  lists  the  personnel  distributions  and  cost 
factors  stored  as  part  of  the  model.  These  may  be  overridden  by  inputting  data 
(Formats  14-16). 

Output  5,  Graduat  Summary  (Figure  A-5)  - Output  5 is  a time-phased  summary  of 
Active  Duty  Force,  Guard  and  Reserve,  other  DoD  and  Non-Dod  graduates.  The  numbers 
reflect  only  those  students  who  actually  graduate  in  that  year.  For  example,  if  the 
course  duration  were  greater  than  one  year,  then  no  graduates  would  appear  in  year  1. 
The  "in  progress"  column  indicates  the  number  of  students  who  entered  in  years  4 and 
5 but  who  will  not  graduate  until  years  6 or  7.  All  Air  Force  students  except  those 
assigned  to  the  Air  National  Guard  and  the  Air  Force  Reserve  are  termed  Active  Duty 
Force.  Guard  and  Reserve  refers  only  to  Air  Force  Guard  and  Reserve  students. 
Students  assigned  to  DoD  agencies  other  than  the  AF,  such  as  the  Army  and  Navy 
(including  their  Guard  and  Reserve  components)  are  classified  as  other  DoD.  Finally, 
students  from  non-DoD  agencies,  including  foreign  governments  (e.g..  Military 
Assistance  Program),  are  termed  Non-DoD. 

Output  6,  Manpower  Summary  (Figure  A-6)  - Output  6 is  a two  part,  time-phased 
summary  of  student  and  base  permanent  party  manyears.  Part  1 is  a functional  break- 
down covering  all  course  personnel  while  Part  2 is  a PCS/TDY  breakdown  limited  to 
AF  personnel . 

Part  1 

Total  Student  Load:  Total  student  manyears. 

Instructors  include  all  academic,  remedial  and  special  requirements  instructors 
as  well  as  course  monitors  and  supervisors. 

Curriculum  Personnel  determine  training  requirements  and  develop  course  written 
materials.  Normally  attached  to  the  Curricula  of  a training  department,  these 
personnel  share  the  task  of  courseware  development  with  the  course's  instructors. 

Hardware  Maintenance  Personnel  are  responsible  for  the  preventive  and  correc- 
tive maintenance  of  course  hardware.  Media  hardware  maintenance  is  generally  the 
responsibility  of  the  Training  Services  Division  (school  level),  while  all  other 
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types  of  hardware  are  usually  serviced  by  the  base  Maintenance  and  Supply  Group  or  a 
specialized  maintenance  squadron. 

Facilities  Maintenance  Personnel  are  responsible  for  the  maintenance  and  repair 
(including  custodial  service)  of  course-related  facilities. 

Training  Administrative  Personnel  perform  training-related  overhead  functions 
not  otherwise  accounted  for.  This  includes  functions  at  the  school  level  (Admini- 
strative Division,  Training  Evaluation  Division,  Operations  Division,  and  Foreign 
Military  Affairs  Office),  at  the  department  level  (Administrative  Section,  Require- 
ments Unit,  and  Instruction  and  Measurement  Unit),  and  at  the  branch  level  (branch 
administration).  It  does  not  include  the  functions  provided  by  the  Training  Service 
Division  or  the  Instructor  Training  Division  (school  level)  or  the  Curricula  Unit 
(department  level),  however. 

Base  Operating  Support  (BOS)  personnel  perform  the  following  base  housekeeping 
and  service  functions:  supply,  transportation,  security  police,  material  staff, 

comptroller,  personnel  staff,  food,  coimmissary,  housing,  laundry  and  dry  cleaning, 
recreation,  education,  transient  aircraft  maintenance,  and  general  (not  course 
specific)  facility  maintenance. 

Medical  Personnel  provide  medical  and  dental  care  to  military  personnel. 

Total  Base  Permanent  Party  represents  those  personnel  assigned  to  a base  for 
the  purpose  of  performing  duty  in  the  furtherance  of  the  mission  of  that  base.  Thus, 
students  are  excluded  from  this  total. 

Total  Course  Manyears  represents  total  student  load  plus  total  base  permanent 
party. 


Part  2 

Active  Duty  Force  PCS  Student  Load:  The  number  of  PCS*  manyears  accrued  by 

Active  Duty  Force  students.  Pipeline  students  are  always  assumed  to  be  PCS  while 
lateral  and  upgrade  students  are  PCS  only  if  the  course  duration  is  20  weeks  or  more. 

Base  Permanent  Party  - AF  Only:  Total  base  permanent  party  less  non-AF  instruc- 
tors. Specific  program  elements  charged  are: 

Specialized  Training 
Air  Force  Instructors 
Curriculum  Personnel 
Hardware  Maintenance  Personnel 
Training  Administrative  Personnel 

♦permanent  £hange  of  Station  means  the  reassignment  of  a military  person  from  one 
permanent  duty  station  to  another.  A-26 
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Hospi tals 

Medical  Personnel  assigned  to  hospitals 
Real  Property  Maintenance  Activities 

Facilities  Maintenance  Personnel  (course-related) 

Facility-related  portion  of  BOS  Personnel  (general  base  housekeeping) 

Base  Operations 

BOS  Personnel  (except  facility-related  portion) 

Medical  Personnel  assigned  to  dispensaries 
Active  Duty  Force  TOY  Student  Load:  Number  of  TOY*  manyears  accrued  by  Active 

Duty  Force  students  including  Program  8 TDY  students.  Pipeline  students  are  never 
TOY  while  lateral  and  upgrade  students  are  TDY  only  if  the  course  duration  is  less 
than  20  weeks.  They  may  be  assigned  to  any  program  element  in  Air  Force  Programs 
1-4  and  6-10. 

Guard  and  Reserve  Student  Load:  Number  of  manyears  (PCS  and  TDY)  accrued  by 

Air  Force  Guard  and  Reserve  students.  All  Guard  and  Reserve  manyears  are  charged 
to  Program  5. 

Total  Air  Force  Manyears:  The  sum  of  total  Program  8 manyears.  Active  Duty 

Force  TDY  student  manyears,  and  Guard/Reserve  student  manyears. 


1 

] 


Output  7,  Courseware,  Hardware,  and  Facility  Requirements  (Figure  A- 7)  - Output  7 
is  a summary  of  key  courseware,  hardware,  and  facility  inputs  and  computer  require- 
ments . 

Output  8,  Functional  Cost  Summary  (Figure  A-8)  - Output  8 is  a breakdown  of  total 
course  costs  by  function. 

Investment  Costs 

Courseware  Procurement  includes  four  separate  costs:  (1)  the  initial  cost  of 

producing  the  master  from  which  all  subsequent  copies  will  be  made;  (2)  the  cost  of 
reproducing  and  packaging  the  required  number  of  copies;  (3)  the  annual  cost  of 
revision  due  to  change  in  course  content;  and  (4)  the  annual  cost  of  replacement  due 
to  loss,  damage,  and  normal  wearout.  It  does  not  include  the  cost  of  courseware 


**Temporary  Duty  means  duty  at  a location  other  than  the  permanent  duty  station  at 
which  a member  performs  temporarily  under  orders  which  provide  either  reassignment 
to  the  old  permanent  station  or  assignment  to  a new  permanent  station. 
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development  such  as  script  writing  and  editing,  which  is  accounted  for  within 
curriculum  manpower  pay  and  allowances. 

Hardware  Procurement  embraces  the  following  cost  elements:  (1)  the  cost  of 

purchasing  the  incremental  number  of  units  required  each  year,  including  initial 
replacement  part  stocks;  (2)  credits  for  any  surplus  items  returned  to  the  Stock 
Fund;  and  (3)  the  annual  cost  of  replacing  those  units  worn  out  during  the  year. 

The  annual  cost  of  replenishment  repair  parts  is  accounted  for  elsewhere.  Negative 
numbers  indicate  a dominance  of  Stock  Fund  credits. 

Facility  Construction  represents  the  total  cost  of  constructing  and/or  modify- 
ing facilities  required  for  course  use. 

Operating  Costs 

Pay  and  Allowances  provides  for  all  officer,  airman,  and  civilian  pay  and 

Ci  vi  1 i an 

Basic  Pay 
Life  Insurance 
Health  Benefits 
Terminal  Leave 
Workman's  Compensation 
Civilian  Retirement 
Overtime 


PCS  Costs  include  the  expenses  incident  to  the  permanent  change  of  station  of 
students  and  instructors:  transportation  of  personnel  and  dependents;  shipment  and/ 

or  storage  of  household  goods;  and  mileage  per  diem  and  subsistence  allowances  while 
in  travel  status. 

TOY  Costs  include  commercial  transportation,  car  rental,  mileage  allowances  and 
tools,  per  diem,  and  incidental  expenses  incurred  by  students  in  authorized  travel 
status. 


allowances.  The  elements  accounted  for  are  as  follows: 
Officers  Airmen 


Basic  Pay 

Special  Pay 

Basic  Allowance  for 
Quarters 

Basic  Allowance  for 
Subsist'jnce 

Uniform  Allowance 

Family  Separation 
Allowance 

Separation  Payments 

Social  Security  Tax  - 
Employer's  Contribution 


Basic  Pay 

Special  Pay 

Proficiency  Pay 

Reenlistment  Bonus 

Basic  Allowance  for  Quarters 

Clothing  Allowance 

Separation  Payments 

Social  Security  Tax  - Employer's 
Contributi on 
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Instructor  Training  is  composed  of  instructor  education  training  costs  (the 
cost  of  the  six-week  instructor  prerequisite  course  on  education  principles)  and 
initial  factory  training  costs  (the  cost  of  providing  the  initial  instructor  cadre 
with  necessary  specialized  equipment  or  system  training). 

Miscellaneous  Operating  Costs  consists  of  three  distinct  elements: 

1.  Computer  Service  Charges:  Any  computer  expenses  incurred  as  a result 

of  using  computer  based  instruction. 

2.  Hardware  Replenishment  Repair  Parts:  The  cost  of  purchasing 

miscellaneous  hardware  repair  parts. 

3.  Miscellaneous  Supplies:  The  cost  of  personnel  supplies  (food,  clothing, 

etc.)  and  general  office  supplies  (paper,  pencils,  forms,  etc.). 

The  planner  should  carefully  inspect  each  type  of  output  to  assure  himself 


that  the  answers  are  reasonable.  Areas  where  large  costs  appear  may  focus  further 


planning  effort  to  reducing  these  costs. 
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